As you hinted at, incompetent employees already make expensive mistakes. Once AI gets to a point where it makes less expensive mistakes, employers would be incentivised to replace the people with machines.

Driving is an easy example, humans crash, AI will still get involved in crashes, but if it is involved in significantly fewer crashes then it would seem almost irresponsible to have humans driving.

I think ultimately it just comes down to me having higher expectations of AI ability than others.

Have you played around with chat gpt? I'd highly recommend it, it's pretty incredible, and a lot of it's limitations are ones that have been intentionally placed on it e.g. it doesn't have access to information from the last year or 2, and there are certain topics it has been restricted from talking about(e.g. race issues and religion).