Viewing a single comment thread. View all comments

PredictorX1 t1_ixlvw4d wrote

Reply to comment by Beneficial_Law_5613 in [D] inference on GNN by Beneficial_Law_5613

Even if your model outputs 0.4 for one class, and 0.6 for the other, at a 93% accuracy, it has learned something. The real question is: How do you intend to use this model's output? If you only care about assorting items to these two categories, then you have achieved 93% accuracy already: Whether this is high enough for your purposes is dictated by the larger problem being solved. If, however, you would prefer more specific predicted probabilities, then you should consider other performance measures, such as cross-entropy.

6

Beneficial_Law_5613 OP t1_ixlx0hc wrote

After training the model I saved the weights. Then I loaded the weight to the model and gave the model a sequence of the data to see/visualize the output of the model on images(basically if the ego vehicle should make a lane change or nor) and even when it should keep the lane change I get a 49% it should change the lane. I am trying to predict if the ego vehicle should make a lane change or not in some data(even if it hasn't been trained on all that data)

−1

jellyfishwhisperer t1_ixlzco4 wrote

The model is doing what you told it to. In that scenario it said keep the lane and it was right. Congrats! You should not think of the outputs as probabilities. They add to 1 but if the model has a score of 0.3 for keep lane that doesn't mean there is a 30% chance it should keep the lane. It's just a score (unless you've built some more sophisticated probabilistic modeling into it)

As mentioned above cross entropy is a good metric. Another metric you may consider is a ROC curve. It will show performance across thresholds. Maybe 0.5 as a cut off isn't best?

And for what it's worth I wouldn't want to be in a vehicle that incorrectly switched lanes 7% of the time ;)

2

PredictorX1 t1_ixlzohu wrote

Good points! I'd also mention that, if probability estimates are desired, the numeric model outputs could be calibrated as a separate step at the end.

2

Beneficial_Law_5613 OP t1_ixm3fpz wrote

Yes but in some cases when the car should keep the lane(or is keeping the lane/its not making any lane change) I get a 76% that it should make a lane change. Thats why I am confused, and for more information: Pred=model(data) PredS = nn.Sigmoid(Pred)*100 Print(PredS) # and here when I give the data of a car that is not making a lane change i get 76% for all its data points/frames. But being honest I don't know if this 76% is for lane changing or lane keeping.

−1

jellyfishwhisperer t1_ixm5zz6 wrote

I'd make sure you know what outputs go with what prediction. Metrics can come after that.

3