You can, but the idea of having a minimum 1/4 inch or 5mm is pretty standard practice, so you can insert 2 wedge shims, one per side to square your frame. Leaving a smaller gap leaves less room for shims and my result in a twisted frame if only 1 shim can be inserted.

It's so shims can be inserted to make sure the door frame is vertical and plumb. Rough openings are exactly that, rough openings, and are rarely perfectly square and plumb. The gap and shims give you room to adjust it so your doorway is square