Good managers drive success by setting reasonable goals and holding people accountable to those. How they meet those goals shouldn't matter. If someone can do the job in one hour and fuck off on Reddit the other 7, that's your failure as a people manager. You need people visible to you because you don't know how to manage work. So yes, that makes you a shit manager, completely unrelated to remote work. That you had 2-3 years to do a better job managing work and you still can't do it without visibility tells me you can't grow as a manager.

Seems like a good thing your manager isn't micromanaging you because you should be on a PIP.