Hi builder here.

The idea that it can only be done in 2-5 years is ludicrous. Sure there's a queue but most projects are slow burns and many homeowners aren't ready to pull the trigger for whatever reason. If you've got the cash and the plans and the land and you're ready to go well have a foundation in the ground as soon as it thaws and depending on the size, you'd be in it within the year.

I know for a fact if you kept it small and simple you'd be out the door for less than 300k. Lumber isn't cheap but it's not the expensive stuff either. The expensive stuff is the windows and the cabinets and the countertops and the appliances and heating/cooling, light fixtures, plumbing fixtures, and all the other things that go into a new home. Plus you've got to pay someone ($60/hr) to put all those things together the right way. A simple uninsulated barn shouldn't cost much more than $125/ft².

Obviously if you're trying to save a buck you can look for deals & help from friends & family with the labor.

Moral of the story: it can be done. But at what cost or with what sacrifice?

There's a saying that somewhat applies here: There's three things everyone wants: cheap, fast, & good. You only get two.