320kbps is completely fine. The quality differences between that and lossless music is not very big even with the most detailed headphones. I have to focus hard in order to hear the differences with my headphones.

Headphones will always be the biggest difference in audio quality/listening enjoyment. Generally when you go from cheap (<$100) to mid priced ($150-$600) is when you'll hear the biggest difference in audio quality. After that it's all diminishing returns, and you'll have to listen to more high end stuff ($800-$1500) to find if you value those smaller improvements. After that higher end stuff, anything more expensive will be less of an improvement and more of finding whatever suits your tastes most.

You also have to remember that different people will like different headphones. Don't take peoples' words as gospel, and know that you'll only know if you'll like a headphone by listening to it with your own ears. I would not have my favorite headphones I have now if I listened to what other people liked the most.

Go to an audio store if possible to try some stuff out. If you're in the US, my personal favorite stores are The Source AV (Torrance, CA), and Audio 46 (New York City). Both of these stores are the biggest in the country, and are absolutely amazing. If you live near, or are on vacation in these 2 areas you should definitely head to either store because you won't regret it.