Why Humans Are Terrible at Evaluating Probability, and What That Means for Product

In February 2013, someone at the CDC pulled up two charts on the same screen. One was the agency’s own weekly count of doctor visits for influenza-like illness, built the slow way. Clinics reporting in, data cleaned, a report published days later. The other was Google Flu Trends, the company’s real-time estimate of flu prevalence, built from search queries and updated instantly. The two lines were supposed to track each other. They didn’t. Google’s line was cresting nearly twice as high as the CDC’s actual count, and had been running hot for months. A tool built to predict a real-world event, at planetary scale, with more data than any epidemiologist had ever touched, was simply wrong, and the reason traces back to the same psychology of probability bias that trips up individual humans making individual bets. That’s the uncomfortable part. It wasn’t a data problem. It was a human one, laundered through code.

Google Flu Trends launched in 2008 to justified fanfare. By correlating search terms with CDC flu data, it could estimate outbreak levels two weeks before official reports existed. A genuine edge in a domain where two weeks is the difference between containing an outbreak and chasing one. For a few years it worked well enough to be cited in Nature as a landmark of what “big data” could do. Then it didn’t. In the 2012–2013 season, it overshot actual flu prevalence by more than 100%. Looking back over a two-year stretch, researchers found it had over-predicted flu prevalence in 100 of 108 weeks. Google quietly retired the public tool in 2015.

What Actually Went Wrong With Google Flu Trends?

The published diagnosis, from a 2014 Science paper by David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani, names two failures: “big data hubris” and “algorithm dynamics.” The second one is the interesting one, because it’s a probability-judgment error wearing an engineering costume.

Google’s model treated search-term frequency as a stable proxy for illness. It isn’t. When flu became a media story (front-page coverage of a bad season, panicked friends texting about it), people searched flu-related terms whether or not they were sick. The searches spiked because the topic was salient, not because the virus was spreading. That’s the availability heuristic at population scale: people, and by extension the model trained on their behavior, mistook how easily flu came to mind for how likely flu actually was. Amos Tversky and Daniel Kahneman described this exact substitution in 1974. People estimate frequency and probability by how readily examples come to mind, not by counting the actual base rate. Google’s algorithm didn’t have a mind, but it inherited the bias anyway, because it was trained on the search behavior of millions of minds that had one.

A predictive model doesn’t need a mind to inherit a cognitive bias, it just needs training data generated by minds that have one. That’s the finding that should unsettle anyone building a product on user behavior data, not just epidemiologists.

Why Did the Algorithm Get Worse the Longer It Ran?

Because nobody was checking it against a stable outside anchor, and the errors were self-reinforcing rather than self-correcting. Google’s system had no mechanism forcing it to reconcile its probability estimate against a slower, boring, less exciting ground truth on any regular cadence. It was optimized to predict search behavior, and search behavior increasingly reflected media coverage of Google Flu Trends’ own prior predictions, plus unrelated winter-season searches (the 2012 update, released right after a bad 2009 H1N1 season, had been tuned on data contaminated by pandemic-driven searching).

This is base rate neglect at institutional scale. Base rate neglect is the tendency to anchor on vivid, specific, immediate signals (a spike in searches, a single unusual data point), while discounting the boring, stable prior rate that should discipline the estimate. Individually, this is well documented. In 1978, researchers Ward Casscells, Arno Schoenberger, and Thomas Graboys asked 60 physicians, house officers, and fourth-year Harvard Medical School students a simple question: a disease has a prevalence of 1 in 1,000; a test for it has a 5% false-positive rate; a patient tests positive; what’s the probability they actually have the disease? The correct answer, using Bayes’ theorem, is about 2%. Most of the doctors said 95%. Only about 18% answered correctly, trained physicians, at Harvard, on a question core to their job, got the actual base rate almost backward.

Google’s engineers were not less capable than Harvard physicians. They were subject to the same bias, at the level of system design rather than individual judgment: the vivid, immediate, high-resolution signal (millions of live searches) crowded out the boring, delayed, low-resolution one (the CDC’s lagged but accurate counts) that should have been anchoring the estimate.

Isn’t This Just a Data Science Failure, Not a Psychology One?

No, and this is the point most retrospectives on Google Flu Trends dance around. The technical post-mortems focus on overfitting, on the 45 search terms the model latched onto that correlated with flu for spurious seasonal reasons (searches for high school basketball, it turned out, correlated with flu season for the boring reason that both happen in winter). That’s a real technical failure. But overfitting itself is what base rate neglect looks like when you give it a training corpus: the model, like the physicians, like most people asked to estimate a probability, gravitated toward whatever correlated vividly with the outcome in the sample it had, and discounted the slower, harder-to-see prior that should have constrained it.

The counterargument is that this is unfair to the humans, that the Casscells study and its descendants are lab artifacts, word problems dressed up as medicine, and that real clinicians with real patients don’t actually reason this badly. The evidence says otherwise, and also says something more useful: the failure isn’t in the human, it’s in the format. Gerd Gigerenzer and Ulrich Hoffrage showed in 1995 that when the identical probability problem is reframed from percentages into natural frequencies, not “a 5% false-positive rate” but “out of every 1,000 people, 50 will test positive without the disease”, the rate of correct Bayesian reasoning roughly doubled. The physicians weren’t incapable of the math. The percentage framing was actively hostile to how the human mind tracks probability; the frequency framing worked with it instead of against it.

That’s not a minor caveat. It’s the whole thesis in miniature. Bad probability judgment isn’t a fixed trait of unsophisticated people. It’s what happens when a probability is presented in a format the brain didn’t evolve to parse, to a person (or a system trained on people) who has no reason to distrust the number they’re looking at.

What Does This Actually Mean for Product?

It means every dashboard, churn score, and confidence badge in your product is a percentage framing wearing a business-intelligence costume, and it’s producing the same failure mode Google Flu Trends produced, at smaller scale, every day.

A churn-prediction model that outputs “this customer has an 82% probability of canceling” is asking a human (a customer success manager, a VP reading a board deck), to do exactly the Bayesian reasoning the Harvard physicians failed. Without the base rate of overall churn, without knowing how the model was trained, without a frequency framing, that 82% gets treated as a near-certainty rather than what it usually is: a noisy point estimate with a wide, unstated confidence interval. Teams then take actions (discount offers, executive check-in calls, resourcing decisions), calibrated to 82%-as-fact rather than 82%-as-estimate-with-error. A/B testing tools have the identical failure built into the “95% significant” badge that half of practitioners still treat as “95% chance this is real,” when it’s actually the inverse conditional. The probability of seeing this data if there were no real effect, not the probability the effect is real. It’s the same confusion the physicians made about the mammogram, replayed in a growth team’s Slack channel.

Recommendation and personalization systems have their own version of Google’s algorithm-dynamics trap: a model trained on what users click will over-index on what’s vivid and recent rather than what’s actually predictive of what they want, and because clicks generate more of the same clicks, the error compounds instead of correcting, precisely the loop that took Google Flu Trends from mildly optimistic to reporting flu at double the true rate before anyone pulled the plug.

What Would Have Actually Changed the Outcome?

Not better engineers, and not more data. Google Flu Trends had both, in abundance, and failed anyway. The Science paper’s authors ran the counterfactual themselves: a model that simply combined Google’s real-time signal with the CDC’s slower, lagged, boring historical data cut the error rate roughly in half, from a mean absolute error of 0.486 down to 0.232. The fix wasn’t a smarter algorithm. It was forcing the vivid, immediate signal to answer to a dull, stable prior on a fixed schedule, instead of letting it run unchecked because it felt more current and more precise.

That’s the one change that would have saved it, and it’s the one most products still refuse to make: build the boring base rate into the interface itself, not as a footnote the reader has to go looking for, but as the number sitting right next to the exciting one, every single time.