He said he’d figure it out. How is that passive aggressive?

You made your stance clear. He needs to organize his own transportation. He has now said he will.

Take your trip and relax! You deserve it! Adding several hours of driving wouldn’t appeal to me either. That doesn’t make you a jerk.