320 CBR is definitely not bad. Lossy encoding is generally quite smart. The bad reputation of MP3s comes from music websites that were encoding 128-190kbps. These were bandwidth-limited. I used to have plenty of home brew OGG and MP3 VBR CD rips that sounded just fine.
There are many expensive and unremarkable headphones anyway and among the good ones, there isn't a very big difference in transparency, even if the price can vary wildly. $150 gets you a trusty hifi pair to live with.