Well, you are missing a couple of things.

#1. The force expended will take the path of least resistance. Which is pushing the bullet out and down the barrel. At which point a portion of the force it released out the end with not resistance.

#2. Concentrated above force is behind a small object. That has little to no resistance and is magnified by all force being directed from behind to push it out of the barrel.

#3. The recoil is only the force that is not expended out of the barrel.

#4. If you plug the barrel, you have violent explosions where the force has no place to release and often kills the person holding the gun because the force is directed equally in all directions.

If you had a firecracker and placed, it on your open palm and lit it. It exploded it would hurt like hell but you might not have massive damage. Lit it and close your hand around with a fist and you are going to loose fingers. The blast from the firecracker dissipates in all directions and you hand only gets a portion of that expelled blast. Close your hand and you hand takes it all. Sort of the same idea.

TRDL: The vast majority of the force produced is concentrated and expelled out of the barrel where there is no resistance. In recoil you are only getting the force that could not be vented out the barrel and in the vest getting hit by the bullet you are getting everything else which is the vast majority of the expended blast translated into kinetic energy.