Definitely didn't use a clear caulk. I'm able to pass a putty knife through to the other side. Yeah I know the pov/depth seems weird but it's just a view from the top of the door. I posted a wider shot for less confusion.

The only air coming out on the top is where the door gasket is not reaching the drywall. I'm 100% sure that the issue is a combination of air coming in through the door as well as the siding of the building.

I'll have to give that adjustment a try. Thanks