Submitted by Beneficial_Law_5613 t3_z3ik75 in MachineLearning
[removed]
Comments
Beneficial_Law_5613 OP t1_ixlvfr7 wrote
Yes the class accuracy is 93%. And yes i agree with you that it can be 0.49 to 0.51. But does this mean the model is not learning? Or that by luck it gets 93% accuracy because after the activation functions it rounds 0.51 to 1, and 0.49 to 0? Or what is the best way to evaluate/inference a model(my model in this case)?
PredictorX1 t1_ixlvw4d wrote
Even if your model outputs 0.4 for one class, and 0.6 for the other, at a 93% accuracy, it has learned something. The real question is: How do you intend to use this model's output? If you only care about assorting items to these two categories, then you have achieved 93% accuracy already: Whether this is high enough for your purposes is dictated by the larger problem being solved. If, however, you would prefer more specific predicted probabilities, then you should consider other performance measures, such as cross-entropy.
Beneficial_Law_5613 OP t1_ixlx0hc wrote
After training the model I saved the weights. Then I loaded the weight to the model and gave the model a sequence of the data to see/visualize the output of the model on images(basically if the ego vehicle should make a lane change or nor) and even when it should keep the lane change I get a 49% it should change the lane. I am trying to predict if the ego vehicle should make a lane change or not in some data(even if it hasn't been trained on all that data)
jellyfishwhisperer t1_ixlzco4 wrote
The model is doing what you told it to. In that scenario it said keep the lane and it was right. Congrats! You should not think of the outputs as probabilities. They add to 1 but if the model has a score of 0.3 for keep lane that doesn't mean there is a 30% chance it should keep the lane. It's just a score (unless you've built some more sophisticated probabilistic modeling into it)
As mentioned above cross entropy is a good metric. Another metric you may consider is a ROC curve. It will show performance across thresholds. Maybe 0.5 as a cut off isn't best?
And for what it's worth I wouldn't want to be in a vehicle that incorrectly switched lanes 7% of the time ;)
PredictorX1 t1_ixlzohu wrote
Good points! I'd also mention that, if probability estimates are desired, the numeric model outputs could be calibrated as a separate step at the end.
Beneficial_Law_5613 OP t1_ixm3fpz wrote
Yes but in some cases when the car should keep the lane(or is keeping the lane/its not making any lane change) I get a 76% that it should make a lane change. Thats why I am confused, and for more information: Pred=model(data) PredS = nn.Sigmoid(Pred)*100 Print(PredS) # and here when I give the data of a car that is not making a lane change i get 76% for all its data points/frames. But being honest I don't know if this 76% is for lane changing or lane keeping.
jellyfishwhisperer t1_ixm5zz6 wrote
I'd make sure you know what outputs go with what prediction. Metrics can come after that.
Tgs91 t1_ixmylf3 wrote
What are the probability outputs during training? What about cross-entropy? Is there a difference between the training cross-entropy and the testing cross-entropy (assuming you are following the same process for testing as you are for inference). Cross-entropy is what actually gets optimized in the training process, and that's what drives probabilities towards 0 and 1. You could perform poorly on cross-entropy and still potentially get a decent accuracy. If your testing cross-entropy matches training and val cross-entropy, then the probability outputs are correct and your model just isn't very confident in it's answers. What kind of regularization are you using? Are you doing label smoothing on the prediction outputs? High regularization or label smoothing could drive your predictions away from 0 and 1.
Is there a class imbalance in your data? Is 93% a significant improvement over a more basic method like KNN? It could be that your model didn't really learn much but gets good accuracy because class imbalance makes accuracy trivial.
PredictorX1 t1_ixluuih wrote
When you say that accuracy is 93%, are you referring to class accuracy? In other words, your model may output a probability, but class accuracy measures how often the predicted probability is on the correct side of 50%. If so, then predicted probabilities pf a 93% accuracy model could certainly frequently be 0.4 and 0.6 (or even 0.49 and 0.51), since the model is still accurately predicting classes.