Having only one set of clothes is inconvenient. Your set of clothes wil not be suited well for all times of the year, you might be hot in the summer but cold in winter too.

As someone pointed out, you'd have nothing to wear while your set of clothes is in the laundry, which, would be quite often if you don't want your clothes to smell.

Different kinds of clothing need cleaning on different cycles. It is common to change underwear every day, t-shirts at least after two days and jeans can sometimes be worn for a week without smelling.

The wear on your clothes is not spread out. Your clothes will break more quickly if you wear and wash them a lot. Especially clothes like jeans can lose a tiny bit of color with every wash and look quite bad after many wash cycles.

If you only had one set of clothes, a laundromat would be wasted, because they usually can't use little enough water for that few clothes, so you would likely need to hand wash.

Socially:

People will notice you wearing the same clothes every day and draw conclusions. Most will, irrespective of your actual laundry habits, assume you walk around in dirty clothes half the time.

In case you are talking about one "selection" of clothes and multiple duplicates of each, that is usually considered weird, but still somewhat fine. If that's what you are after, I'd advise you to buy two different styles of pants and shirts so you can look somewhat unique between your outfits and still keep the simplicity.