Maybe? But if it is it will likely be one of the last things automated. General purpose robotics that are humanoid or can do all the same tasks as a human are likely the most complicated to make. With that being said no one truly knows where AI and therefore robotics will be in 20 years and it's definitely possible.

I wouldn't worry about job security though, if that were to happen we would likely have had UBI or something for a long while due to other mass automation (trucking, manufacturing, generative design, creative, etc...).

Even if you can't make everything on site, the cost of shipping and resources will still be just the cost of those robots and energy (which would also be built up by robotics and if we had that level of robotic production it's likely that fusion has been built out so energy would be near free), and then the cost to make those robotics would also be the cost of the robots that built them.

If robots build robots which can then build everything else including general contractor work (aka general purpose robotics) without the need for any human bottleneck then you start this absurdly powerful compounding growth trend that drives the effective cost of anything to near 0. The only limits would be land and raw materials. Land could be optimized if labor is effectively free by building vertically (vertical farms, lab grown meat, etc...). Raw materials could be mined from asteroids if we have said level of full automation and fusion propulsion driven reusable spacecraft (it's unlikely we would do this otherwise since the cost to get said materials is prohibitively expensive compared to just mining them on earth).

So could a robot at one point become general enough to do all the work of construction? Of course it can. will that happen in 20 years? No one knows. Should you be concerned? Probably not since at that point the whole economy would have to be rewritten and well we'd have plenty of abundance for everyone.