falldeaf t1_jeehm0d wrote on March 31, 2023 at 12:59 PM

Reply to comment by Relevant_Ad7319 in Language Models can Solve Computer Tasks (by recursively criticizing and improving its output) by rationalkat

I bet it will be possible with the multi modal version! Essentially just give it access to the ability to take screenshots and an API for choosing mouse position. It'd be interesting to know if that could work in a one shot fashion.

WonderFactory t1_jeg6rye wrote on March 31, 2023 at 7:51 PM

It's too slow at inference for something like that. It's probably far easier to do it the other way around. If you want your software to interface with GPT 4 build in some sort of scripting interface to your app

falldeaf t1_jeg8xaf wrote on March 31, 2023 at 8:05 PM

It would be slower, but I'd disagree that it's too slow for that to work. In fact, I bet it could write something like autohotkey scripts to accomplish what it needs to do. You wouldn't have to have video and slowly move your mouse across the screen. You could get a screenshot, figure out where to move the mouse, then move the mouse to those coordinates and press left mouse button, take a screenshot to confirm the app is open, etc.

Having said that, anything that can be accomplished by opening a terminal should just be done there as it would be faster. In the short term though, there's lots of applications that are designed for humans that it would be great for LLM's to be able to interface with. Maybe in the long term they'll just write their own applications to accomplish something we'd normally need a gui for. Maybe there will be interfaces that have a human viewable component but most of the controls will gone. Like imagine a 3D modelling application that just has a viewer with just a few buttons to move the view around (It'll be easier to just spin the object to an angle yourself then say it.) But you'll have pointing and painting tools to help collaborate with the AI. ::draw a circle around a part of the mesh:: Make this area a little rougher. ::point to a leg, then draw a line coming out in a curve:: Have a tooth-like spike come out right here. Etc.

It'll be neat to see where this all goes, I suspect that UIs will radically change but in the near-term I'm sure there will be stop-gaps using current tech, too.