You need to understand that the "pivot point" of a jigsaw is right in front of the blade. Many people try to force the front where they want it to go, instead of stopping forward movement and moving the back end of the saw.

That being said, open ended blade tools will wander, you'll never get a band saw like cut

You could attempt to screw a scrap piece of wood inside the radius of the cover plate, we could give you something to screw to.

Is your issue that the hole is so big that the handle almost falls through it?

This shouldn't be necessary. The handle usually is sandwiched between the door with machine screws. Only the latch plate on the edge of the door is screwed in.

Top or bottom hinge might be set too far forward or back.

Something is out of plumb then. Compare plumb of both sides of jamb.

Your strike plate is too far forward. Move it back a quarter inch, you may need to drill/chisel into the jamb to allow for the bolt to seat into the strike plate hole properly

Yeah bro let me just sand down a painted wall

Is this before painting?

If the humps are minor, feather out with drywall compound and a 12 inch trowel. If significant, you might need to scrape

Any texture differences well disappear after priming and painting

Do not use grout or caulk. Install a shoe mould to the floor.

Grout will crack. Caulk is going to look dirty and awful after a month.