Comments
SupportstheOP t1_jef9yvd wrote
Faster and better gains in AI research --> Better AI systems --> Faster and better gains in AI research --> Better AI systems
And then there we have it.
Relevant_Ad7319 t1_jee0rb7 wrote
Can someone explain how this can work? How does chat gpt know where to click on a computer?
SkyeandJett t1_jee2yc5 wrote
I don't want to stay that's trivial but it is easily solved. However that's more or less irrelevant. GUIs are for humans. GPT accesses things directly through a CLI API. This paper more or less confirms what everyone else has been saying and experimenting with. GPT-4 might not be AGI, but enhanced with memory, chain of thought, task generation and prioritization, self-checking and correction, etc. it probably is. Now give it access to tools, things like TaskMatrix coming soon and frankly it becomes an extremely powerful autonomous agent. You tell it what you need and it just...does it. This is all going to come together very quickly. Then drop an immensely more powerful core into the system, i.e. GPT-5 and things start getting stupid.
Itchy-mane t1_jeece6h wrote
I literally sold all my agix coins after seeing taskmatrix. Shit looks revolutionary when paired with gpt 4
Relevant_Ad7319 t1_jee3l13 wrote
But not everything has an API. I think we need GPT to simulate mouse and keyboard inputs like a human in order to automate everything what a human can do on a computer
EDIT: No idea why I get downvoted for this đ¤ˇââď¸ This sub is strange
falldeaf t1_jeehm0d wrote
I bet it will be possible with the multi modal version! Essentially just give it access to the ability to take screenshots and an API for choosing mouse position. It'd be interesting to know if that could work in a one shot fashion.
WonderFactory t1_jeg6rye wrote
It's too slow at inference for something like that. It's probably far easier to do it the other way around. If you want your software to interface with GPT 4 build in some sort of scripting interface to your app
falldeaf t1_jeg8xaf wrote
It would be slower, but I'd disagree that it's too slow for that to work. In fact, I bet it could write something like autohotkey scripts to accomplish what it needs to do. You wouldn't have to have video and slowly move your mouse across the screen. You could get a screenshot, figure out where to move the mouse, then move the mouse to those coordinates and press left mouse button, take a screenshot to confirm the app is open, etc.
Having said that, anything that can be accomplished by opening a terminal should just be done there as it would be faster. In the short term though, there's lots of applications that are designed for humans that it would be great for LLM's to be able to interface with. Maybe in the long term they'll just write their own applications to accomplish something we'd normally need a gui for. Maybe there will be interfaces that have a human viewable component but most of the controls will gone. Like imagine a 3D modelling application that just has a viewer with just a few buttons to move the view around (It'll be easier to just spin the object to an angle yourself then say it.) But you'll have pointing and painting tools to help collaborate with the AI. ::draw a circle around a part of the mesh:: Make this area a little rougher. ::point to a leg, then draw a line coming out in a curve:: Have a tooth-like spike come out right here. Etc.
It'll be neat to see where this all goes, I suspect that UIs will radically change but in the near-term I'm sure there will be stop-gaps using current tech, too.
CommunismDoesntWork t1_jef7r37 wrote
Unix adopted the philosophy that text is the ultimate API, which is why everything on Linux can be done through the CLI, including moving the mouse. And LLMs are very good at using text. So everything sort of does have an API.
Relevant_Ad7319 t1_jefsg9h wrote
Oh thatâs cool I didnât know that
arckeid t1_jeegitn wrote
I think this is a good way no just to make the AI, but to help humans to stay in sync, for me it's looking the advancements are already so fast.
[deleted] t1_jeer8ve wrote
[deleted]
SgathTriallair t1_jeerghs wrote
The task paper addressed this. If it can see the screen then in hasn't cases a keyboard and mouse API will be the best option.
How it knows where to click on the screen is that it is trained to understand images just like it understands text. So it will know that a trash can means you want to delete data the same way we know that.
[deleted] t1_jeea351 wrote
[deleted]
CaliforniaMax02 t1_jeetc75 wrote
There are a lot of tools which solve complex mouse and keyboard tasks and processes manually (UiPath, Blueprism, Automation Anywhere, etc.), which can be interfaced to this.
They can automatically open email attachments, copy texts, open an Excel (or any other) window, and enter the text structurally, etc.
Relevant_Ad7319 t1_jeftc4u wrote
It should be able to switch from doing taxes, browsing the web, and playing valorant within minutes just like a human can do. Thatâs not possible with UI path etc.
Sure in theory you can find/write an API for every task that you want it to do but for me thatâs not what an AGI is
basilgello t1_jee2lyt wrote
Just like Generative Asversarial Networks operate: there is a creator layer and a critic layer that hope to reach a consensus at some point. As for "how does it know where to click": there is a huge statistics made by humans (look at page 10 paragraph 4.2.3). It is a specially trained model fine-tuned on action task demonstrations.
Relevant_Ad7319 t1_jee3ah6 wrote
Task demonstrating in form of screen recordings? It says their approach only needs a few examples but Chatgpt doesnât even work with videos as input right?
basilgello t1_jeecmqt wrote
Correct, GPT4 is not meant to accept videos as input. And probably not screencasts but explained step-by-step prompts. For example, look at page 18 table 6: it is LangChain-like prompt. First, they define actions and tools and then language model puts the output which is actually high-level API call in some form. Using RPA as API, you get mouse clicker based on HTML context. Another thing HTML pages are crafted manually, and system still does not understand the unseen pages.
SgathTriallair t1_jeertpn wrote
Given that it can accept images, they may be able to shoehorn videos in. The next version we use as a base will need multi modality equal to humans (i.e. all of our senses) in order to relocate all of what we do.
Jeffy29 t1_jeg33to wrote
That's one of the most boring names I've have seen a paper have lol, though I skimmed it and it looks quite good and is surprisingly readable. Though I don't think this method will be adopted anytime soon, from the description it sounds quite heavy on inference and given how much compute is needed for current (and rapidly growing) demand, you don't want to add to it when you can just train a better model.
The current field really reminds me of the early semiconductor era, everyone knew that there were lots of gains to be had by making transistors in a smarter way but there wasn't the need when node shrinking was going so rapidly and gains were, it wasn't until the late 2000s and 2010s when the industry really started chasing those gains which there are plenty but it isn't nearly as cheap or fast as the good ol' days of transistor shrinking. But it is good to know that even if LLM performance gains inexplicably completely stops tomorrow, we still have lots of methods (like this and others) to improve their performance.
rationalkat OP t1_jedzrqf wrote
An excerpt from the Paper:
"6 Broader Impacts:
Although the results presented in this paper are only on a research benchmark, if we extrapolate forward the capabilities of these models and methods, we anticipate vast broader impacts that have the potential to revolutionize numerous industries. By allowing LLMs to execute tasks on computers, our approach can enhance the capabilities of AI assistants and automation tools. This could lead to increased efficiency, reduced labor costs, and improved user experiences across any sector which uses computers to do work. We are most excited about gains in productivity in science and education, including AI research, which will lead to even faster development of new beneficial technologies and treatments."