For years, I used a glass water bottle with a silicone protective outer layer for the very reason you described. Starbucks used to sell them.

Also, I do have one stainless steel suggestion. I currently use an Artic water bottle, and it's been great. It doesn't taste like metal at all.