DisasterEquivalent t1_jdk10wf wrote
Reply to comment by BinarySplit in [D] I just realised: GPT-4 with image input can interpret any computer screen, any userinterface and any combination of them. by Balance-
I mean, most apps have accessibility tags for all objects you can interact with (it is standard in UIKit) - The accessibility tags have hooks in them you can use for automation. so you should be able just have it find the correct element there without much searching.
Viewing a single comment thread. View all comments