When it comes to statistical measurements, the standard accepted error margin for a new result to be considered a legitimate discovery is very small, but it's still possible for the result to be outside that range by sheer chance.

For example, imagine an experiment that examined cancer rates in connection to smoking said that there was only a 1 in 1 million chance that smoking did not increase chances of getting cancer, and all seeming connections were just a coincidence. That's a very, very low chance of it being unrelated, but it's still possible, and 1 in a million chances happen every day.

If a second experiment is done, using an unrelated dataset, and it also finds the same thing at the same chances, it greatly reduces the chance of the first dataset supporting that conclusion by sheer fluke. It's still not completely impossible, but the chances of both experiments being flukes is exponentially lower than just one of them being one.

Suppose you are doing research on jelly beans and their effect on acne.

Suppose also there's actually no effect.

A group of scientists does a study, and finds no effect. Since there's no effect, they don't publish their study.

Around the world, maybe many scientists are doing research on the link (if any) between jelly beans and acne. Maybe it's the color? One group studies purple jelly beans, finds no link, and doesn't publish. Another studies red jelly beans, finds no link, and doesn't publish.

Then one day, just by chance, a group found a relationship that was significant at the 5% level.

This was inevitable, since so many groups of scientists are independently studying the phenomenon, in ignorance of what others are doing.

So now there's a published paper linking green jelly beans to acne.

Even more scientists start doing similar research. What other colours have an effect? Do green jelly worms also "cause" acne?

Since there's a lot of research now, more articles get published - red jelly worms are linked to acne, with a p value of 0.02. Green chiffon cake is linked to acne, with a p-value of 0.03. Nobody publishes the results that show no link.

Eventually, the literature shows a strong relationship between confectionery and acne, especially green, especially with gelatin. Food scientists, dermatologists, regulators rely on this information to provide professional advice and to draft laws. It Science journalists inform the public of this new threat to teen health. Soon "everyone knows" how dangerous green food colouring is...

... But actually no link exists.

If people took the time to replicate the studies, and published the failed replications, this wouldn't happen.

Making the initial paper insist on a stricter level of proof doesn't help, because the whole problem is that negative results aren't being published, and the literature is showing a biased set of results. It would be better to publish the results of every study, so people could see whether that 5% result is something that stands alone, suggesting some real link between two things, or if it's just one of a whole series of similar studies, most of which showed no relationship between the things at all.

To oversimplify...

In studies you are looking for what is known as statistical significance, which is basically shorthand for it being very likely that the effect you are seeing is a real effect rather than just being an unlikely chance effect.

If you are looking at the effects of a drug, perhaps the effect that you are seeing is just random chance - the people who took the drug just randomly got lucky and the people who didn't take the drug got unlucky.

So you do replication to rule out that chance. If you do two independent drug trials and they both show the same effect, the chance that it is due to random fluctuations is much smaller.

Batch effects, for one. Something could be wrong about a particular batch of some reagent so that it causes aspecific effects in your assay. You then generate measurements that, sure enough, reach statistical significance. If enough of your publication hinges on that bad data, it could even cause a retraction.