I think this is quite an important and fundamental question. Of course the answer will depend on the task. But in theory, what deep learning is doing is maximum likelihood. That is, minimize the average error. Doing "average" on the "whole task" is superhuman most of the time.

I think the context in which the task is performed, and what is meant by "outperform", is important. If given enough time and energy, a person could probably find all the dogs in a dataset of images. But could they find 200k dogs in a dataset of millions of images overnight? Probably not. In this sense, machines far outperform humans who are limited by attention span and ability to parralelize.

Can a knife cut better than a human hand? After all, it was made by human hands... Yes, AI can be designed and trained to outperform humans at any task that we can frame as a ML task. The big advances have come from clever ways to frame tasks in a way that ML can work on it.

I think the best way to answer your question is to ask you to be more precise about what, exactly, you mean by "outperform".

There's some limited sense in which your reasoning works as you seem to have envisioned. A generative model like GPT or GANs is typically built at least partly to produce output that's indistinguishable from what is produced by a human, using some kind of autoregressive data set or adversarial objective. By definition, it cannot do better at that goal, because a human has a 100% success rate, by definition, at producing something indistinguishable from what is produced by a human.

But there are limitations to this reasoning:

1. Producing any arbitrary human-like output is not actually the goal. People don't evaluate generative models on how human-like they are, but rather on how useful their results are. There are lots of ways their results can be more useful even if they aren't quite as "human-like". In fact, the motivation for trying to keep the results human-like is mainly that allowing a generative model too much freedom to generate samples that are very different from its training set decreases accuracy, not that it's a goal in its own right.
2. That's not all of machine learning anyway. Another very common task is, for example, Netflix predicting what movies you will want to watch to build their recommendations. Humans are involved in producing that data, but it's not learning from data about what other humans predicted users would watch. It's learning directly from observed data about what humans really did watch. Such a system isn't aiming to emulate humans at all. Some machine learning is even trained on data that's not generated by humans at all, but rather the objective it's training to optimize is either directly observed and measured, or directly computed.
3. Even in cases where a supervised model is learning to predict human labeling, which is where your reasoning best applies, the quantity of data can overcome human accuracy. Imagine this simpler scenario: I am learning to predict which President is on a U.S. bill, given the denomination amount. This is an extremely simple function to learn, of course, but let's say I only have access to data with a rather poor accuracy rate of 60%, with errors occurring uniformly. Well, with enough of that data, I can still learn to be 100% accurate, simply by noting which answer is the most common for each input! That's only a theoretical argument, and in a realistic ML context it's very difficult to get better-than-human performance on a supervised human-labeled task like this. But it's not impossible.
4. And, of course, if you look at more than just accuracy, ML can be "better" than humans in many ways. They can be cheaper, faster, more easily accessible, more deterministic, etc.

AI will surpass humans in all domains where it can generate problem solving data. AlphaZero did it. Trained in self-play and beat humans. No imitation, no human data at all.

What we need is to set up challenges, problems, tasks or games for the language model to play at. And test when it does well, and add those solutions to the training set. It will be a loop of self improvement by problem solving. The learning signal is provided by validation, so it doesn't depend on our data or manual work. It can even generate its own challenges.

More recently AlphaTensor found a better way to do matrix multiplication. Humans tried their hand for decades at this task, and in the end the AI surpassed all of us. Why? Massive search + verification + learning = a "smart brute forcing" approach.

Yes, ML can outperform humans in certain tasks.

1. Quantity can sometimes make a very big difference - if you could sit down and train a human on the same amount of data, the human might be on par with ML...but that's often not possible

2. Training data is not always generated by humans.

3. Given the same data, there are connections or perspectives that humans have not followed or considered.

It absolutely can, but it depends on the task.

When it comes to questions of human estimation, there is the Wisdom of the Crowd effect where individual people can either over- or underestimate a value, but the average of many predictions often is extremely accurate. You can either train a system directly on the average of many human predictions or you only have a single prediction for each sample and during training the errors tend to average out.

And even for tasks where there is a definite ground truth available that doesn't depend on human estimation it can happen. For example in many medical applications the diagnosis is provided by the lab, but physicians often need to make a decision whether it is worth it to take a biopsy and send to the lab. In this case they often are taught heuristics: A few features by which they decide whether to have a sample analyzed.

If you take samples and use the groundtruth provided by the lab, it is not inconceivable that a deep learning based classifier can discover additional features that are not used by physicians that lead to better predictions. Obviously, it won't get better than the lab results.

As you migth have guessed, I work in the medical field. And we have done the evaluation for one of our products: It is not able to outperform the top-of-the-line experts in the field. But it is easily able to outperform the average physician.

From a very theoretical point of view, we can imagine a knowledge "A" useful for a task A, a knowledge B useful for a task B. That's how humans would apply their knowledge. But we could teach a model to learn knowledge A (or A+B) and apply it to task B, and it would eventually perform better.

Humans don't have all the knowledge and don't apply everything they could know to every tasks perfectly. Models also aren't perfect but they could do more combinations and perform better on certain tasks because of that.

But I can take another exemple. Here is a task: "a human says N images contain dogs and M images contain cats, the model must reproduce this behavior". Would a perfect model designed to exactly reproduce the human be able to outperform a human on this task? No. The human would make mistakes, and the model would reproduce these mistakes. But we don't design or train our models to exactly reproduce what a human did, that would be a risk of overfitting, we use regularizations so that even by reproducing what humans did a model can do better and not reproduce some mistakes.

Yes, in general. As long as the human is accurate most of the time, the ML model will take the errors as outliers and outperform the human. The AI is generally finding trends in the data and then using those trends to predict future points. As long as the human tagging is accurate enough to get the true trend across, the AI can outperform them. It's important to understand that an AI model will perform DIFFERENTLY than a human, for better or worse, since it uses different processes. Some mistakes that a human makes, the AI would not make, but there are mistakes a human might not make that the AI will make.

Yes, I believe there are 2 factors playing a role here:

1. Models could potentially correct some errors of the human labeler using their generalization power, provided that the model is not overfitted.

2. You should differentiate between outperforming a human and outperforming humans. Labels usually represent the collective knowledge of a number of people not just one.

What does "truly outperform humans" mean? It sounds so broad, some kind of philosophical question, like "How many angels can dance on the head of a pin?". What are you asking? Can a machine truly outperform a human at climbing, hammering, flying, calculating, sorting, or drawing accurate conclusions in a limited domain given a certain input? Of course, obviously. Can it truly outperform a human at falling in love, tasting an apple, or getting drunk on wine? No.

Humans have always been augmented by their tools. That is one of the fundamental characteristics of being human. At the tasks they're designed for, artificial tools vastly increase the performance of humans, and allows them to outperform what they could do without it. Humans have enhanced their cognitive abilities with all kinds of calculating and "thinking" machines, for millennia. A human with a clay tablet and a reed can far outperform other humans at remembering things. But what is a clay tablet - or a PC, or anything - without humans? Nothing, as far as humans are concerned.

OP, you should watch the machine learning street talk YouTube channel and watch whatever videos interest you. It is an incredible resource from top minds all around the world in the field. Only caveat is i would say don’t start with the Noam chomsky episode. It is a great episode, but a bad first episode to introduce yourself to the series because so much exposition goes into explaining the extreme unfortunate technical issues they faced in fixing a messed up recording.

I’ll tell you what i think you are asking then answer. Fwiw i am NOT an ML engineer. I think you are asking “can machine learning models with current technological limits outperform humans in any given task considering the data is possibly compromised by human error? What about in the future?”

Firstly, self supervised learning approaches are apparently beginning to perform better than human guided models in many tasks. I’m referring to isolated cognitive tasks which can operate within a VERY limited amount of specified parameters. Take diffusion image generation. There are perhaps 60 to 80 settings you tweak. before you include token processing (processing the prompt. Smth like 4 words is more or less equivalent to a token) . My point here is that these are GAN driven, and in a way, while the data is not cleaned, the really poorly fitting data will be statistically reflected in output probably to the extent that it is an outlier. So low quality things will be output less because within the context of the model, the model may “understand” this is not a desired output.

Second, while your question is basic on the surface, it is actually a major subject of debate still. My personal opinion is that there are major structural components and AI architectures that still need to be created, if not to imitate human conscious thinking, executive function, attention, strategizing, and possibly desire and more nuanced reward systems, and they must all be integrated in such a way that when they train themselves on data that they are able to discover best practices for multi-step, multimodal, and various cognitive approaches before it is as intuitive as talking to another person and explaining what you want and getting the desired result.

While transformers (an ai architecture invented in 2017) are powerful and appear to learn data and it’s semantical importance, and can lead to better performance than humans in many tasks, there is still something “dumb” about these models, namely that they are highly specialized. This is why the datasets are absolutely ginormous, yet if you put them outside of their speciality they have no clue what the hell is going on really.

There are actually some interesting counterpoints to this. For instance, for google imagen, a diffusion image generator, i believe it has one of the largest parameter datasets for image generation. What is particularly interesting though is that even though the model is trained on images, because it has seen so much text in images, has learned to spell spontaneously. Therefore you can ask for an image of a guy holding up a sign that says “a computer wrote this sign” and it will create images of each letter in order to appear as the words requested.

While that is incredibly interesting, datasets will be eventually approaching the size of the internet itself and only able to do simple tasks like this. As a tiny human, i didn’t need to absorb the entire world too learn about its generalities. Ultimately i think that is the answer you are looking for.

I personally believe that consciousness has to do with integrated multimodal information processing and that intelligence is simply a brain having ample raw data stored in memory, but in extremely efficient and compressed ways structured from general to specific. It is less like the information is stored there as discrete facts, and more like the information is stored in configurations of layers of processing, so that multiple different concepts can be represented within these spaces at any given time.

One very strong support to this idea is considering what human attention actually is. I think attention is less an accessing of raw data and “focusing” on that data than it is a holistic reconstruction of a concept from multiple constituent components. This is especially why it would be nearly impossible for a person to think in great depth about two very different and unrelated concepts simultaneously. However This is also why metaphors and analogies appear so powerful as explanatory devices for humans, because a metaphor takes two simple but seemingly unrelated concepts and makes them fit together in a way that makes sense within the given context of the metaphor. We understand that a metaphor is not literal though, which is the only reason they work, and is why even high functioning autistic people may have difficulty understanding them, because their “attention” in sensory processing and therefore concept representation is not focused enough and gets processed more as a broad and heavy multimodal concept that is hard to parse because it is taken as literal data.

My point though is that current machine learning models, while they have layers of processing, still are behind in general and broad intelligence because they do not have multiple integrated data types and models consulting with one another to form more generalized notions of concepts. They only have these concepts in particular forms of data by themselves which in turn makes them error prone no matter the data.

I don’t think it is bad data that is the problem as much as missing context to allow the model to understand what is and isn’t bad data

Yes. First of all the ai can learn from all humans not just one. And the end result could be an ai with the knowledge and skills of 4 billion people. Which would ofcourse outperform anyone on the planet simply because no person can do it all...

But then ai has also been used already to advance mathematics. Solving previously unsolved mathematical problems.

Here's an example:

https://www.sciencealert.com/ai-is-discovering-patterns-in-pure-mathematics-that-have-never-been-seen-before/amp

It already does. There are many complex tasks which can't even be performed by humans. (e.g. time-series forecasting). The relationships between variables is in a lot of cases so complex that humans simply can't find them on their own. But, we can use ML as a black-box approach to perform these tasks.

Where ML completely blows humans out of the water is when it has some kind of continuous learning scheme. Well known examples are GANs and active learning. Another major technique right now is to use reinforcement learning schemes to improve pre-trained models. By creating another model that gives a preference between two outcomes (original model and finetuned model), a reward can be computed to further improve the model. This is why ChatGPT is so good. Definitely look up Reinforcement Learning from Human Feedback to learn more about this.

I think Karpathy said this in a tweet ? Or was it LeCun ? Someone talked about it though ...

Think of it more like this: the total intelligence/capability of all humans greatly exceeds the total intelligence/capability of any individual human. When we train ai models, we are imbuing them with capabilities derived from many humans. But once produced, they are easily copied so we could quickly have a population of them, where each individual ai is equivalent to many humans (and thus greater than any individual human) but maybe less capable than all humans. Eventually the population of enough such ais is more capable than the population of humans.

And then, if the ais can train new generations from prior ais, this pattern could repeat and explode.

[removed]

Are you asking if it can outperform a single human or a collective of humans?

Can a machine learning model determine the make and model of a Toyota better than I can? Yes. Is there a human on the planet that can better determine the male and model? Yes.

We should have a more rigorous definition of "outperform". What are we comparing? Your question touches on the idea of internal versus external validity - if the data is fundamentally flawed, there is performance ceiling if it doesn't reflect the use case of the ML algorithm developed using it. It may be internally valid (the ML model is trained correctly), but has poor external validity (the ML model doesn't apply to the task it was trained for).

It’s a really important question. It surely can for certain tasks. I think the general reason why models trained in a supervised setting can outperform us, even though we are the ones that tell them all the right answers, is because we don’t tell them how to get to those answers. When models learn to find the answers via some novel method/logic that is better than ours it can use that logic to outperform us.

sometimes it is possible, like AlphaZero for example.

Yes eg a medical image classifier can outperform the average doctor. This can be verified by having 10 doctors review each image in the test set to establish a true-consensus.

Even when ML is trained with noisy labels, there are techniques to obtain robust models whose accuracy is overall better than the noise-level in each label. One good opensource library for this: https://github.com/cleanlab/cleanlab/

Another way to get ML that outperforms individual data labelers is to have multiple annotators label your data. Crowdlab is an effective method to analyze such data: https://cleanlab.ai/blog/multiannotator/

So there are a few things to consider here. What exactly does it mean to "best humans"? In some cases, we can train on data from professional or highly skilled humans, and use that to outperform the average human.

In other situations, we can train an ML model to be a sort-of average of a lot of humans, getting rid of a lot of their individual errors and thus performing better than the average.

In other situations, we may have access to data that is not directly generated by humans, or it's made by humans with access to more data than what the model has access to. For example, to train a vision based ML system for autonomous driving, you could use ground truth data obtained by other means (e.g. LIDAR).

Another sort-of similar situation is training an ML model to predict the stock market. Here, you can use actual stock market trends as the ground truth, rather than the guesses of humans.

So all in all, there are many situations where ML models aren't limited to human capabilities.

Google search, AlphaZero… don’t we have examples of AI all around us that are better than humans?

Also isn’t this the same as asking “if a human learns from other humans can they never be smarter than the people around them”?

One of the most interesting things about ML to me is that it can exploit large amount of data (even not entirely correct) to reach super-human level.

Suppose you trained a classification model on a large amount of images. Some of these images will be probably labelled wrong by humans (e.g. because are blurred, the object is not entirely visible, undecision between two classes, etc.). Other similar images will be labelled right, perhaps by another human. But since the model saw many many images it can average its understanding of the images. Keep in mind that this is not anymore true if you have biased training data, i.e. the error is always done in the same way.

https://www.nature.com/articles/s41746-019-0112-2

TL;DR AI is better at detecting and scoring prostate cancer than expert pathologists themselves.

Let me just post this as an example: https://www.youtube.com/watch?v=kBFMsY5ZP0o

A model trained to detect humans through walls using bouncing wifi signals, based on teacher-student training.

Even though all the original data has been generated by humans, I feel pretty confident in claiming it probably outperforms us at estimating body position from wifi signals.

This obviously transfers to other domains as well- there's certain types of features our brains are wired to pick up(vision, sound, proprioception), some types it is capable of learning(written language, math), and plenty it isn't though to be fair we're pretty bad at most of our learnable tasks- we read and calculate ridiculously slowly, and our ability to break that work down into chunks and memorize intermediate results is the only thing that saves us.

Compared to that, a DNN is a tabula rasa ready to learn how to extract arbitrary features independent of how our cognition of them works, without having to jump through repurposing brain networks to decipher non-native stimuli the way our brains do.

Not all AI is ML.

In closed systems, AI can emulate how the brain is forward-looking.

For example in a game like chess or go, there isnt data to be dissected, rather there is sheer force of compuation.

The monte carlo tree algorithim can simulate games based on the specific state of a board. Humans can do that however to a much lower degree and less deep.

You should look into some work by Jaron Lanier about this, he sometimes argues that AI is just the theft and encapsulation of human abilities.

Yes, but I'm less sure about language models at the really high level (eg arriving at novel solutions to hard problems through LLMs).

Most ML in practice isn't about doing better than a person, it's about doing it faster and cheaper. Could a human who studied my viewing habits curate better Netflix recommendations for me? Obviously, but Netflix can't afford to do that for everyone and it would take forever.

There's also ML that's not based on data generated by humans. I know we're in the era of LLMs, but that's not all there is

Depends on how your labels, or more generally the target data distribution, is generated. If it’s generated by human subjectivity yea I would agree. However it’s not hard to think of situations where the labels are the output of a physical process or, in the case of prediction, a future event. In these cases the label is not ambiguous and thus not subject to human interpretation. You also have RL systems that play games against each other and reach super human performance that way. Read about AlphaGo or AlphaStar for example.

>Of course, in certain areas, like speed of computation or accuracy, computers will also be better than humans

I'm very confused what else do you expect it to do ? That's all ML is ; a formalized, structured way of categorizing data. The only reason it's better than human is because it's so much faster that it can't possibly be performed by an human.

>can it truly outperform humans?

Again I'm confused, because if speed isn't the concern, at least accuracy indeniably is. And ML already outperforms humans accuracy-wise.

The calculators are already outperforming humans. I bet you can't compute 567 times 891 in 1 second. They can.

Adding to the chorus, but I would have to say yes. Like you, I have hardly any background in ML, so I'm just going off of what our ML team presented to my team (since we're the ones on the front lines with customers)...

In the real estate world - I think there is honestly just too much data for a human to go through. Our company was estimating how long we think it would take a house to sell, and were training the ML model based on local historical data. Going through all of that data, for what I think was the past 30 years, and then making an estimation for every property? No way a human could keep up. The accuracy compared to a human estimation I can't really speak to, but with how quickly the real estate world moves, speed is the number one here.

I'd be interested to hear other takes on specific industries. Are there any that you guys work in that you feel humans>ML?

Once you have a model that performs as well as a human in some domain, you can then use reinforcement learning to get it to perform better. Of course doing this is easier in some places than others.

Due to the amount of knowledge that exists and humans limited time and speed to learn there is an upper cap at how much performance a human can achive, yet a computer isnt limited by this in the same way.

Imagine a medical analisys taking into consideration all data from multiple exams and checking every medical study in existance is something humans could never do, simply because the amount of data the exams bring in is far too much and there is far more medical knowleadge already discovered than a human could learn. Hence a machine with all that knowledge would outperform any human doctor. (Obvious problem is building such a machine, but we have already studies of ML doing increadible analysis based on x-ray scans)

One risk is that the human knowleadge migh end reaching a cap since there isnt a incentive to have humans push the limits given that the models are already better than them. (One of the big problems that will likely be caused by image and text generating models, so we will actually have soon the results in a decade or two with how they influenced new artists, teachers and students)

If ML is based on human data, can it outperform humans? Yes. HINT: scale.

{Search, Recommendation, Ads} systems are something we all use today. These ML systems greatly outperform humans. Most of these successful ML systems rely on human-generated data to train. For example, Google looks at what users clicked to train their relevance models. Facebook uses which ads get the most dwell time to learn what types of ads to show next time. Reddit uses user upvote/downvote data and user clicks to learn which posts to boost.

Peter Norvig who ran search quality at Google stated that getting above 80% recall for these systems was quite good [reference: https://qr.ae/pryCgm]. The average human performance on most of these tasks is around 90%. Most of these systems are outperforming humans even though they are not getting high enough recall on individual samples.

Why?

Since these things operate at scale, not every suggestion has to be perfect. The user can ignore the bad suggestions. Furthermore, in the more advanced versions of these systems (i.e. personalized versions of these systems), one could improve recall by simply learning about the user.

Most ML systems that have a defined goal and scale will be able to outperform humans. They will outperform humans even if they use human-generated data. This will only be true if they perform at scale.

A more interesting version of your question would be can we build a single system that can outperform all humans in all tasks? This is the AGI question.

I think with an ability to understand, it would be able to outpace humans. In a sense outperform, so I'd say yes. I find the research being done (I think by Nvidia) on recovering color and shape data from sparse images really interesting. It goes beyond what I'd imagine possible.

There is a youtube channel, TwoMinutePapers. Highly recommend it

Besides the computational speed advantage mentioned by OP and other human-related aspects such as attention span / human fatigue, I think there's a bit more nuance in the answer depending on what "generated by humans" means.

​

Take a binary image classification example, where the task is to classify whether an image is a dog or not a dog. Here, the labels are "generated by a human" by looking at the same input that the algorithm receives. In this case, assuming all labels are correct, I would argue that machine learning is unable to achieve a higher accuracy than the human labeler ignoring the previous factors mentioned, since the "correctness" of the classification was defined by the human itself. If the human was in peak condition and without time constraints, the human labeler should be able to achieve 100% accuracy all the time.

​

On the other hand, suppose the task is to perform time-series prediction, where the label is obtained from the future. Even though the dataset was collected by a human, the "generation process" is not from a human labeling the data but rather by some other process. In this case, machine learning has the potential to outperform humans.

It will outperform sdome humans. I.e: some shitty lawyer who cant go beyond the surface will be surp assed. Those whounderstand subtelties wont have to worry

Id say an ML model can only hope to be as good as its training data.

If the person generating that data just looks through tons of pictures and marks them as apples or not then the model will probably be about as accurate. But if the person choses pictures they’re certain have apples in them or not from some other method (like taking pictures of apples and things that definitely arent apples) then the model may do much better than people can, since its training data is perfect.

I think that no-one properly answered your questions. All of the answers above talk about the quantity and how ML model can find milion dogs on milion images per night. But this is obvious and you also mention it in the post. ML models can outperform us in the speed of computation or accuracy as you say, but they can not outperform us in a truly general case as you say. At least not now.

Supervised learning can never do better than humans since, the best ML solution will have the same accuracy as the humans.
We would need something like universal problem solver, which is able to solve more and more complex problems over time and learn from it. But it seems that not so many people are working on this general AI.

Machines doesn’t seem to think rationally and logically as we humans do. I’m not exactly sure about the exact research in this direction, but maybe it is possible to stimulate this rational/logical thinking by set of rules/discrete maths etc? (Like I’ve heard there are many theorem solvers which exists) But again it should be coded by humans right? it’s like expecting machines to be conscious of themselves, which at the current stage they aren’t.

have been the idea of a bot. Haha

Absolutely. Because it's never just the input from humans- presented with an image and a label for it given by a user, the model is not limited to learning only the relationships that the human used to generate the label- the image is right there, after all. So when all goes well, the model can learn relationships in the data that humans are unable to because the human labels are used to guide learning on the source material.

Additionally, there are many ways to allow a model to treat labels as 100% true (i.e. the word of God) but allow for some incorrect labels. In which case, it's entirely possible for the model to do better than the human(s) did even on the same data.

Of course something trained on humanS can possibly outperform a human. Is it a mystery to you that a tug of war between humanS on one side and a human on the other will have a known outcome?

Less facetious example. People have created 'averaged' faces (using machine learning) across all men and women and also across men and women of certain races. Is it a surprise that these 'averaged' faces are more attractive than the average person?

Yes, it can, because it can gather it's own data.

If a house is built out of parts of trees, can that house ever stand taller than those trees used to be?

If the human errors are random then yes, if there is a consistent pattern that allows the model to learn the "correct" errors then that can cause problems

Sounds like a god of the gaps argument. Machines may not be better at humans for any given task today, but it seems to be moving pretty quickly in that direction. With that said, it seems humans become a lot smarter with the aid of machines and vice verse.

If kings eat food harvested by slaves, could they win a 1-1 fight against one?

[removed]