Even if no one has tried them on, factories and warehouse are filthy places, and as another poster mentioned, most fabrics and clothing have been treated with chemicals to preserve them.
I rarely wear anything before it's been washed with detergent, though usually use a short cycle for new clothes.

Source: BS in Textiles and Clothing, worked in clothing industry for 5 years, professional seamstress for over 40 years