I've been the kid in this situation.

I can't tell you what's best because I don't know any of you or the details of your situation, and these things affect everyone differently, but my advice is to make him leave.

They love him because he's their dad, kids almost always love their parents simply because they're the parents, at least until they are able to fully realize how poorly their parent treated them. I suspect it will be much the same for your kids.

Based on what you wrote, he doesn't treat them right. He may love them in his own way, as you said, but that doesn't matter in the long run. What matters is how his actions affect those around him, regardless of his own feelings. Keeping him around those kids is going to chip away at their self esteem and could lead to many other issues, like self esteem issues.

I'm sorry for your situation, there really isn't a perfect solution. You just have to pick the best option, and I sincerely hope you choose the kids well being. Not just for their sakes, but yours as well.