How to normalize data which contain positive and negative numbers into 0 and 1

Submitted by Constant-Cranberry29 t3_ywu5zb in deeplearning

I have a dataset that contains negative and positive values. then here I use MinMaxScaler() to normalize the data to 0 and 1. but because the normalized data has negative and positive values in it, the normalization is not optimal, so the resulting prediction results are not optimal. then I try to change the negative data to positive with abs() then the result from abs() is normalized using MinMaxScaler() the result will be better. is there a way for me to keep the negative and positive values but have good predictions?

my last activation function is Sigmoid

Here my model structure:

    model = Sequential()
    model.add(LSTM(64, activation='relu',  return_sequences= False, input_shape= (50,89)))
    model.add(Dense(32,activation='relu'))
    model.add(Dense(16,activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss = 'mse', optimizer=Adam(learning_rate=0.002), metrics=['mse'])
    model.summary()

​

https://preview.redd.it/ymk5w17ukb0a1.png?width=779&format=png&auto=webp&s=b7fe642a0fa18608e2082ec82d8790a701e792c8

Here my code of normalization with abs():

    df = pd.read_csv('1113_Rwalk40s1.csv', low_memory=False)
    columns = ['Fx']]
    selected_df = df[columns]
    FCDatas = selected_df[:2050]
    FCDatas = abs(FCDatas)
    
    SmartInsole = np.array(SIData[:2050])
    FCData = np.array(FCDatas)
    Dataset = np.concatenate((SmartInsole, FCData), axis=1)
    
    scaler_in = MinMaxScaler(feature_range=(0, 1))
    scaler_out = MinMaxScaler(feature_range=(0, 1))
    data_scaled_in = scaler_in.fit_transform(Dataset[:,0:89])
    data_scaled_out = scaler_out.fit_transform(Dataset[:,89:90])

The result using abs():

​

https://preview.redd.it/xk412lchxe0a1.png?width=888&format=png&auto=webp&s=4a227f12591e5dbcb1f5b55da89a0264424cae0f

Here my code of normalization with without abs():

    df = pd.read_csv('1113_Rwalk40s1.csv', low_memory=False)
    columns = ['Fx']]
    selected_df = df[columns]
    FCDatas = selected_df[:2050]
    
    SmartInsole = np.array(SIData[:2050])
    FCData = np.array(FCDatas)
    Dataset = np.concatenate((SmartInsole, FCData), axis=1)
    
    scaler_in = MinMaxScaler(feature_range=(0, 1))
    scaler_out = MinMaxScaler(feature_range=(0, 1))
    data_scaled_in = scaler_in.fit_transform(Dataset[:,0:89])
    data_scaled_out = scaler_out.fit_transform(Dataset[:,89:90])

The result without abs():

​

https://preview.redd.it/o4au3lr3xe0a1.png?width=888&format=png&auto=webp&s=974e8b8b1fd6e1400c766e6a4615759b308d683b

0

Comments

You must log in or register to comment.

pornthrowaway42069l t1_iwlbxq2 wrote

You can try neglog,

x > 0: log(x)

x<0: -log(-x)

1

Constant-Cranberry29 OP t1_iwlwjy8 wrote

I have try, then it doesnt work.

1

pornthrowaway42069l t1_iwmr91a wrote

emmm... 2 parameter box-cox transform? If that doesn't work as well, maybe there the problem is with something else, between neglog and 2 parameter box-cox you should get decent normalization I feel.

1

Lexa_21 t1_iwlyig7 wrote

You can try to transform your negative values into the [0, 0.5] interval and your positive values into the [0.5, 1] interval.

1

Hamster729 t1_iwmtbs9 wrote

It is not clear what you are doing, because your code does not match your plots. The model in your code outputs values in 0..1 range, but your plots have large positive and negative values. To help you, we would need to understand what exactly is going on. I want either the complete model or the physical significance of your data. Generally speaking, unless signs in your data have no significance (so e.g. a +5 and a -5 correspond to the same fundamental physical state), applying an abs to the data would only make the model perform worse.

1

Constant-Cranberry29 OP t1_iwnx2yp wrote

if you looking from the number why that is not 0-1 because before plotting the value I already transform it to original value.

1

Hamster729 t1_iwny1c8 wrote

Can I see the code that does the reverse transform, in the case without abs?

1

Constant-Cranberry29 OP t1_iwnyhip wrote

df = pd.read_csv('1113_Rwalk40s1.csv', low_memory=False)

columns = ['Fx']]

selected_df = df[columns]

FCDatas = selected_df[:2050]

SmartInsole = np.array(SIData[:2050])

FCData = np.array(FCDatas)

Dataset = np.concatenate((SmartInsole, FCData), axis=1)

scaler_in = MinMaxScaler(feature_range=(0, 1))

scaler_out = MinMaxScaler(feature_range=(0, 1))

data_scaled_in = scaler_in.fit_transform(Dataset[:,0:89])

data_scaled_out = scaler_out.fit_transform(Dataset[:,89:90])

1

Constant-Cranberry29 OP t1_iwnz3zj wrote

I have edit the pictures which contain normalization data

1

Hamster729 t1_iwo4ma4 wrote

Okay. So, as I understand, your labels are usually either zero (before normalization), or negative, and, very rarely, they are positive.

With the abs, it's easy for the model to reproduce the "baseline" level, because it's still zero after normalization, and as long as the last Dense produces a large negative number, sigmoid turns that number into zero.

I think it would work even better if, instead of abs, you set all positive labels to zero, then normalize. (After normalization, the "baseline" level will become 1, also easy to reproduce).

In both cases, the model will work for data points that originally had negative or zero labels, but it won't work for data points with originally positive labels.

You have a problem without normalization, because the "baseline" level no longer 0 or 1 and your model needs to converge on that number. I think it would get there eventually, but you'll need more training, and probably learning rate decay (replace the constant learning rate with a tf.keras.optimizers.schedules.LearningRateSchedule object, and play with its settings.)

The question is, do you want, and do you expect to be able to, reproduce positive labels? Or are they just random noise? If you don't need to reproduce them, just set them to zero. If they are valid and you need to reproduce them, do more training.

P.S. There are other things you could try. Here's an easy one. Drop the abs, drop the normalization, and change the last layer to: model.add(Dense(1, activation=None, use_bias=False))

1

Constant-Cranberry29 OP t1_iwo6ukg wrote

>Okay. So, as I understand, your labels are usually either zero (before normalization), or negative, and, very rarely, they are positive.
>
>With the abs, it's easy for the model to reproduce the "baseline" level, because it's still zero after normalization, and as long as the last Dense produces a large negative number, sigmoid turns that number into zero.
>
>I think it would work even better if, instead of abs, you set all positive labels to zero, then normalize. (After normalization, the "baseline" level will become 1, also easy to reproduce).
>
>In both cases, will work for data points that originally had negative or zero labels, but it won't work for data points with originally positive labels.
>
>You have a problem without normalization, because the "baseline" level no longer 0 or 1 and your model needs to converge on that number. I think it would get there eventually, but you'll need more training, and probably learning rate decay (replace the constant learning rate with a tf.keras.optimizers.schedules.LearningRateSchedule object, and play with its settings.)
>
>The question is, do you want, and do you expect to be able to, reproduce positive labels? Or are they just random noise? If you don't need to reproduce them, just set them to zero. If they are valid and you need to reproduce them, do more training.

I have try using tf.keras.optimizers.schedules.LearningRateSchedule object, it still doesn't work

1

Constant-Cranberry29 OP t1_iwo6vm1 wrote

initial_learning_rate = 0.02

epochs = 50

decay = initial_learning_rate / epochs

def lr_time_based_decay(epoch, lr):

return lr * 1 / (1 + decay * epoch)

history = model.fit(

x_train,

y_train,

epochs=50,

validation_split=0.2,

batch_size=64,

callbacks=[LearningRateScheduler(lr_time_based_decay, verbose=2)],

)

1

Hamster729 t1_iwo99fy wrote

That's a very odd looking time decay rule, and I'm almost certain that it does not do what you expect it to do.

Try:

def lr_time_based_decay(epoch, lr):    
   return lr*0.95

(also see my suggestion from the edit to my previous post)

1

Constant-Cranberry29 OP t1_iwoc176 wrote

still the same even I drop abs, drop normalization, and change last layer to model.add(Dense(1, activation=None, use_bias=False)) it doesn't work

1

sqweeeeeeeeeeeeeeeps t1_iwnu8yt wrote

You are misinterpreting what “normalizing” is. It converts your data to fit a standard normal distribution. That means, you have positive and negative numbers centered around 0. This is optimal for most deep learning models. The interval [0,1] is not good because you want some weights to be negative as certain features negatively impact certain results.

1

Constant-Cranberry29 OP t1_iwnx4ze wrote

so what should I do for solving this problem?

1

sqweeeeeeeeeeeeeeeps t1_iwnx6pv wrote

What’s your problem? Normalized data is good.

1

Constant-Cranberry29 OP t1_iwnxdah wrote

I want reduce the shifting prediction if I not use the abs()

1

sqweeeeeeeeeeeeeeeps t1_iwnxlbc wrote

? Not sure. Train it longer, lower learning rate, are u using teacher forcing? I’m not very familiar with best LSTM practices.

1