This seems interesting. I'll give this a shot. Thanks!

Same initialization but not the exact weights. However, I've run the experiments enough times with the same result for me to be sure that the initial weights aren't an issue.

The simpler model had lower training loss with the same number of epochs. I tried training the second model until it had the same training loss as the first model, which took much longer. The validation did not improve and had a slight upward trend, which I know means that it's overfitting.

I could add more linear layers and based on my experiments it would probably help but my intention is to compare my new model with the old one for which I presume the architecture should be as close as possible.

I wouldn't call it a bigger network necessarily. The second network has two more output neurons compared to the first. Rest are the same. How much difference that makes. Idk

Yes, I trained until the validation loss stopped improving, and then some more just to make sure.

That's what I thought but I haven't been able to get the second model to even match the performance of the first one. I tried regularization methods without much success.

I did. The second model performed worse. I didn't think that was possible.