Book summary “Superforecasting”: summary

Introduction

You face complex decisions about the future while listening to confident experts who routinely fail. When geopolitical conflicts, market shifts, and technological disruptions occur, you need reliable foresight rather than persuasive rhetoric. Yet you often find that prominent commentators offer contradictory forecasts without tracking their accuracy over time.

Philip E. Tetlock discovered that average expert judgment is roughly as accurate as a dart-throwing chimpanzee. Through the massive Good Judgment Project, the author tracked thousands of ordinary volunteers who dramatically outperformed professional intelligence analysts. Rigorous measurement, probabilistic thinking, and structured teamwork allow ordinary people to dramatically outperform professional experts at predicting global events.

This book reveals how to cultivate foresight as a measurable skill. You will learn to recognize cognitive traps, update beliefs using numerical probabilities, and structure teams for maximum accuracy.

• How do you measure predictive accuracy without falling for hindsight bias?
• What mental habits separate elite forecasters from ordinary commentators?
• Can you train yourself to think in probabilities rather than certainties?

1. The Illusion of Expertise and the Nature of Predictability

Global events often seem unpredictable, but rigorous measurement and probabilistic methods reveal that foresight is a cultivatable skill rather than a mysterious gift.

You might assume that prominent experts possess deep insights into the future. From 1984 to 2004, Philip Tetlock conducted a comprehensive twenty-year study on expert judgment. The average expert is roughly as accurate at forecasting global events as random guessing or a dart-throwing chimpanzee. Experts on political and economic questions performed poorly over longer time frames, falling to the level of random chance three to five years out. This finding was often distorted into a total dismissal of all forecasting.

The Dart-Throwing Chimpanzee

Celebrity pundits make unfalsifiable claims because they use vague verbiage without probability guidelines. Tom Friedman of the New York Times was selected for high status and geopolitical relevance while offering claims that are difficult to pin down. Pundits use words like may, could, or might without quantitative definitions. These semantic dances allow them to claim credit when events happen and dodge blame when they do not.

The Optimistic Skeptic

We live in a world where predictability and unpredictability coexist, making it possible to see into the future if we apply rigorous methods. While extreme events can cascade from minor triggers due to nonlinear chaos, many routine aspects of daily life and specific domains remain highly predictable. The author adopts an optimistic skeptic stance, rejecting both naive predictability and total nihilism.

Mohamed Bouazizi setting himself on fire in Tunisia on December 17, 2010 sparked the Arab Spring, illustrating how a single minor event can trigger massive global ripples. Bouazizi died on January 4, 2011, and Tunisian dictator Ben Ali fled on January 14, 2011.

The Necessity of Measurement

Improvement in forecasting requires setting a clear goal of accuracy, measuring results rigorously, and systematically revising methods. Most fields fail to improve because consumers of forecasts do not demand evidence of accuracy, meaning performance is never measured or revised. Without feedback loops, skills cannot develop. Sunrise and sunset forecasts remain reliable across a fifty-year time horizon because meteorologists use a forecast-measure-revise procedure.

In 2011, Barbara Mellers and Philip Tetlock launched the Good Judgment Project, sponsored by IARPA. The multiyear forecasting tournament demonstrated that 20,000 intellectually curious laypeople who tried to figure out global issues could achieve superior predictive accuracy on high-stakes global questions. Ordinary volunteers organized into the Good Judgment Project significantly outperformed professional intelligence analysts and academic competitors. The Good Judgment Project beat the official control group by sixty percent and seventy-eight percent in years one and two respectively, and beat university-affiliated competitors by margins ranging from thirty percent to seventy percent. Bill Flack, a retired Department of Agriculture worker in Nebraska, successfully answered hundreds of difficult geopolitical questions with high accuracy.

2. Illusions of Knowledge and Thinking Systems

For centuries, medicine remained unscientific and dangerous because practitioners relied entirely on strong conviction and intuition rather than rigorous experimentation. You can trace this uncritical certainty through a long history of treatments that were largely useless or harmful.

When George Washington fell ill in 1799, his physicians bled him relentlessly, gave him mercury to cause diarrhea, induced vomiting, and applied hot cups to raise blisters, a hideous regimen that had been standard practice for centuries.

Archie Cochrane fought against medical arrogance by introducing randomized controlled trials, exposing how institutional medicine resisted scientific validation. He argued that the medical establishment suffered from the God complex, relying on the illusion that their personal judgment alone revealed the truth.

Archie Cochrane was once mistakenly diagnosed with terminal cancer by an eminent specialist and wrapped in bandages after having a muscle removed, blindly accepting the verdict without waiting for the pathologist's report.

Thinking About Thinking

Human decision making relies on a dual-system model where fast, automatic System 1 generates instant impressions and slow, effortful System 2 evaluates them. System 1 operates continuously in the background, executing automatic perceptual and cognitive operations without conscious awareness to deliver rapid snap judgments.

When asked how much a ball costs if a bat and ball cost $1.10 and the bat costs a dollar more, people instantly blurt out ten cents from System 1 before System 2 calculates the correct answer of five cents.

System 1 operates on the rule that what you see is all there is, forcing the brain to invent coherent stories even when evidence is completely missing. To deliver rapid conclusions, System 1 treats available evidence as sufficient and reliable without wondering if information is flawed or missing.

In Michael Gazzaniga's research with split-brain patients, when a patient's right hemisphere was shown a snowstorm and pointed to a shovel, the left hemisphere smoothly confabulated a reason, claiming a shovel was needed to clean out a chicken shed.

Bait and Switch

Human nature resists doubt and embraces confirmation bias, grabbing the first plausible explanation while actively dismissing conflicting evidence. Instead of testing hypotheses by trying to disprove them, people look for supportive information and become motivated skeptics when contradictory facts are presented.

Following the 2011 Oslo car bomb and mass shooting, many commentators immediately rushed to the conclusion that radical Islamists were responsible based on past terror attacks, before learning the perpetrator was actually a homegrown right-wing extremist named Anders Breivik.

Human minds rely on fast automatic heuristics that generate overconfidence, requiring slow deliberate analysis to overcome cognitive biases.

When confronted with a difficult question, the mind unconsciously substitutes an easy question as a proxy without you realizing the switch. This attribute substitution is the core mechanism behind cognitive heuristics like the availability heuristic.

3. Keeping Score and the EPJ Study

Accurate forecasting requires precise numerical probability scales, rigorous scoring rules, and flexible thinking styles like foxes rather than hedgehogs.

Judging Forecasts

Forecasts often fail to be testable because they rely on ambiguous terms and unstated time frames. When forecasters omit clear definitions and explicit schedules, observers can retrospectively interpret vague statements to mean whatever they want. This lack of precision protects reputations at the expense of accountability, making it impossible to objectively determine whether a prediction was right or wrong.

Steve Ballmer predicted in 2007 that the iPhone would not get significant market share, but his failure to define significant or specify the exact market and time frame made his quote impossible to judge conclusively. Similarly, an open letter to Ben Bernanke warned that quantitative easing risked inflation and currency debasement without specifying a time frame or quantitative thresholds.

Intelligence analysts and government officials routinely use phrases like serious possibility or fair chance, which producers and readers interpret with wildly different numerical odds. Sherman Kent demonstrated this confusion and proposed establishing designated numerical ranges for specific terms to enforce metacognition and clarity.

CIA analyst Sherman Kent discovered that an intelligence estimate calling an attack on Yugoslavia a serious possibility was interpreted by analysts to mean odds ranging from 80-to-20 to 20-to-80. Later, President Kennedy relied on a fair chance assessment for the Bay of Pigs invasion without realizing the military writer had odds of 3-to-1 against success in mind.

You can apply these standards by following specific rules:

• Reserve the word possible for important matters where analysts must make a judgment without reasonably assigning a probability, spanning from almost zero to almost 100 percent.
• Narrow the range of estimates whenever possible.
• Use designated numerical meanings for terms, such as 93 percent for almost certain, 75 percent for probable, 50 percent for chances about even, 30 percent for probably not, and 7 percent for almost certainly not.

Scoring Forecasts

Because individual probabilistic forecasts cannot be validated by single events, researchers must aggregation many forecasts over time. Calibration measures whether events happen as often as their assigned probabilities dictate, while resolution measures how decisively a forecaster assigns high and low probabilities rather than hedging.

A meteorologist predicting a 70 percent chance of rain every day can only be evaluated by tabulating a massive track record over months to see if rain indeed falls 70 percent of the time on those days. Brier scoring properly incentivizes forecasters to report true beliefs by penalizing overconfidence and mismatch with reality.

Thinking Styles

Philip Tetlock tracked 284 professionals who made roughly twenty-eight thousand predictions on political and economic topics over a twenty-one-year period. The study found that expertise, higher education levels, and access to insider information did not correlate with better predictive accuracy.

Participants included academics, government officials, and media commentators making predictions that spanned one, five, and ten years out, concluding just as the Soviet Union dissolved and Mikhail Gorbachev appeared in commercials.

Predictive accuracy depends on thinking style, dividing thinkers into rigid hedgehogs and flexible foxes. Hedgehogs organize their thinking around a single Big Idea, forcing complex problems into preferred templates and dismissing contrary evidence with high confidence. Foxes draw on multiple analytical tools, shift mental gears, weigh probabilities rather than certainties, and readily admit mistakes.

Larry Kudlow, a prominent economic commentator who applied supply-side economics to predict a continuous Bush boom, insisted there was no recession even as the Great Recession began and Lehman Brothers filed for bankruptcy.

4. Superforecasters and Institutional Blind Spots

Ordinary volunteers in tournaments outperform intelligence analysts by focusing on process accuracy, resisting randomness, and understanding regression.

You often fall for a bait-and-switch fallacy by judging past decisions based on eventual outcomes rather than contemporary evidence. Robert Jervis analyzed the National Intelligence Estimate of 2002 regarding Iraqi WMDs and concluded the intelligence community's judgment was sincere and reasonable because Saddam Hussein acted like someone hiding something, even though the factual conclusion was completely wrong.

Evaluating Intelligence Failures

Bureaucracies rely on process accountability rather than tracking actual accuracy. The Central Intelligence Agency provided analysts with manuals on cognitive biases written by Richards Heuer, but the agency never actually tested whether this training improved forecast accuracy. The intelligence community employs twenty thousand analysts and runs on an annual budget of roughly 50 billion dollars.

The IARPA Tournament

The Intelligence Advanced Research Projects Activity funded high-risk research and sponsored massive forecasting tournaments. Jason Matheny and Steve Rieber of IARPA met researchers Barbara Mellers and Philip Tetlock in Berkeley in the summer of 2010 to launch a massive tournament testing new methods for geopolitical forecasting. The project required a 20 percent margin of victory over the control group in the first year and a 50 percent margin of victory over the control group by the fourth year.

Researchers recruited 3,200 volunteers who passed initial psychometric tests and started forecasting in the first year. Doug Lorch, a retired IBM computer programmer from Santa Barbara, volunteered for the Good Judgment Project, made roughly one thousand forecasts in his first year, and achieved a Brier score of 0.22, ranking fifth among 2,800 competitors. Volunteers received a 250 dollar Amazon gift certificate at the end of each season.

Skill Versus Luck

Humans lack an intuitive feel for randomness and mistake lucky streaks for innate skill. Ellen Langer conducted experiments where thirty times students were asked to watch and predict a rigged coin-toss series, and those who received early hits developed a higher opinion of their own predictive skill.

Regression to the mean helps you distinguish skill from luck. Michael Mauboussin analyzed the balance of skill and luck in his book The Success Equation, noting how human height correlations between fathers and sons demonstrate regression toward the mean with a correlation coefficient typically found of 0.5. Individual superforecasters in the tournament achieved a year-to-year correlation score of 0.65.

5. Supersmart Decompounding and Active Open-Mindedness

High intelligence and deep knowledge help you forecast, but they hit a threshold beyond which they add little value. Testing volunteers showed that regular forecasters scored higher than about 70 percent of the population, while superforecasters scored higher than about 80 percent.

Superior foresight relies on breaking problems into sub-components, starting with statistical base rates, and treating beliefs as hypotheses.

Sandy Sillman, a retired atmospheric scientist with multiple degrees from Brown, MIT, and Harvard, achieved a Brier score of 0.19 and became an overall tournament champion. Extreme IQ and elite degrees remain unnecessary for superior foresight.

Fermi Estimation

Complex estimation problems can be solved by breaking them down into smaller sub-questions, a method known as Fermi-izing. You separate the knowable from the unknowable and produce much more accurate estimates.

Physicist Enrico Fermi famously asked his students to estimate the number of piano tuners in Chicago without using the internet. They calculated 50,000 pianos, 2 hours of time per tuner, 1,600 annual working hours, and a final estimated number of 63 piano tuners.

• Identify the core question to be answered.
• Break down the question into the specific sub-information required to solve it.
• Set a confidence interval representing a range you are 90 percent sure contains the right answer.
• Make reasonable estimates or approximations for each sub-question.
• Assemble the guesses using simple math to calculate a final estimate.

Outside View First

You must always start with the statistical base rate of a broader class before diving into the specific details of a case. People naturally gravitate toward the inside view because engaging details allow for compelling storytelling.

When asked whether the Renzetti family has a pet, people invent stories based on names, whereas superforecasters begin with the outside view that roughly 62 percent of American households own pets. The pet ownership rate for American single-family detached homes is 73 percent.

• Find the base rate or historical frequency of the event within a broader class.
• Use this outside view percentage as your initial starting point or anchor.
• Examine the specific details of the case to see how they adjust the baseline probability up or down.

Active Open-Mindedness

Active open-mindedness treats beliefs as hypotheses to be tested rather than treasures to be guarded. You actively cultivate intellectual diversity and seek out opposing viewpoints rather than gravitating toward like-minded sources.

Doug Lorch created a customized database of hundreds of information sources tagged by ideological orientation, subject matter, and geography. This reading habit constantly forced him to encounter diverse perspectives.

• Take into consideration evidence that goes directly against your existing beliefs.
• Pay active attention to those who disagree with you rather than those who agree.
• View changing your mind as a sign of strength and intellectual growth rather than weakness.

6. Probability for the Stone Age and Information Age

Moving beyond primitive binary thinking requires embracing granular numerical probabilities, high numeracy, and epistemic uncertainty.

Human beings naturally rely on primitive two-setting or three-setting mental dials. You inherit ancestral survival mechanisms where quick binary responses were necessary. You prefer confident answers and struggle with true probabilistic thinking. In Mark Bowden's book The Finish, President Barack Obama and CIA officers debated the identity of the man in the Abbottabad compound. This led the president to declare the situation fifty-fifty. Intelligence analysts expressed confidence levels ranging from 30% to 90% regarding Osama bin Laden's presence in the compound. People often use a 50% probability setting as a stand-alone placeholder for uncertainty rather than a precise mathematical point.

Superquants

Superforecasters possess strong numerical literacy, which helps them process data and evaluate risks. You will find that they rarely rely on complex mathematical models for everyday forecasting. While superforecasters are almost uniformly highly numerate and comfortable with numbers, they rarely deploy explicit arcane math models. Their superior numeracy underpins careful thought, subjective judgment, and nuanced evaluation of relevant information. Cornell mathematics professor and superforecaster Lionel Levine relies almost entirely on subjective judgment. He explicitly aims to prove he could forecast well without using math models.

Probability for the Information Age

Modern science demonstrates that uncertainty is an ineradicable element of reality. You must recognize that absolute certainty is an illusion. Leon Panetta captured this mindset during interviews by repeatedly stating that nothing is one hundred percent. Former Treasury Secretary Robert Rubin and his deputy Larry Summers frequently grew frustrated when top policymakers treated an 80% probability as a guarantee. People consistently misinterpret probabilistic statements, translating high probabilities into guarantees and viewing probabilistic forecasters as incompetent or lazy.

Uncertain Supers

Fine-grained probability assessments directly correlate with forecasting accuracy. Regular forecasters tend to restrict their estimates to tens. Superforecasters frequently utilize single percentage points or fine increments. Superforecaster Brian Labatte naturally breaks down proportions into precise metrics, such as estimating a 65/35 ratio between nonfiction and fiction reading. Research by Barbara Mellers proves that rounding these granular forecasts down reduces accuracy.

7. Belief Updating and Bayesian Reasoning

Superforecasters update beliefs in small increments like intuitive Bayesians, avoiding both stubborn underreaction and erratic overreaction.

Belief Updating

Devyn Duffy made 2,271 forecasts on 140 questions in a single season. Good updating requires cognitive labor similar to the initial prediction process. You can follow this structured practice when approaching complex forecasts:

• Unpack the question into components
• Distinguish between knowns and unknowns
• Adopt the outside view followed by the inside view
• Explore similarities and differences with the views of others
• Synthesize all views into a single vision
• Express judgment using a finely grained scale of probability

Cognitive Biases

Underreaction often stems from belief perseverance where people rationalize away facts. During World War II, California Attorney General Earl Warren oversaw the internment of 112,000 Japanese Americans and maintained this stance even when no sabotage occurred. The dilution effect causes you to overreact or lose confidence when exposed to clearly irrelevant information. In a 1989 study, participants given study hours plus irrelevant details like playing tennis lost confidence in their GPA predictions. You must navigate between Scylla and Charybdis by avoiding both stubborn inertia and erratic swings. Doug Lorch shifted his Arctic sea ice forecast from 55 percent to 95 percent based on a single, month-old report.

Bayesian Reasoning

Tim Minto won the top spot in the third season of the IARPA tournament with a Brier score of 0.15 by making adjustments averaging 3.5 percent. Superforecasters act as intuitive Bayesians who gradually narrow uncertainty through repetitive processing. Political scientist Jay Ulfelder evaluated Chuck Hagel's confirmation by starting with the historical base rate of 96 percent. He adjusted this to 83 percent after factoring in a poor committee performance, two weeks before Hagel was confirmed. Barbara Mellers tested superforecasters on urn problems, demonstrating that they approach optimal mathematical updating much more closely than regular forecasters.

8. Perpetual Beta and Team Dynamics

Maintaining a growth mindset, cultivating grit, and structuring teams with constructive confrontation eliminate groupthink and drive elite performance.

Carol Dweck distinguishes between a fixed mindset and a growth mindset. You embrace challenges and learn from mistakes when you view abilities as malleable rather than immutable. Mary Simpson joined the Good Judgment Project with a growth mindset after missing the 2007 financial crisis.

Top forecasters treat their knowledge base as an unfinished program in perpetual beta. Elizabeth Sloane volunteered for the Good Judgment Project while battling brain cancer and undergoing chemotherapy.

Teams and Group Dynamics

President Kennedy reformed his advisory team following the Bay of Pigs disaster. You can eliminate groupthink using structural safeguards:

• Designate intellectual watchdogs to pursue every bone of contention.
• Set aside protocol and hierarchy to enable freewheeling discussions.
• Bring in occasional outsiders to provide fresh perspectives.
• Leave the room at times to let the group talk freely without executive pressure.

Teams must avoid personal attacks or vague expressions of disagreement. You can engage in precision questioning without causing emotional friction by applying concrete rules:

• Do not make personal shots or say "You're silly" when disagreeing.
• Avoid merely stating "I don't think so" without explaining the reasoning.
• Avoid overly vague questions like "What do you mean?".
• Zero in on specific terms by asking questions like "What do you mean by this term?" or "What evidence supports this claim and over what time frame?".

Superteams and Performance

Superteams aggregate dispersed knowledge more effectively than standard markets by fostering a pro-social culture. Individual superforecasters achieved a 50% boost in accuracy when placed in structured superteams. In tournament trials, superteams consistently outperformed prediction markets by 15% to 30%.

9. Leadership Dilemmas and Scope Sensitivity

Effective leadership under uncertainty requires mission command, which combines clear strategic goals with decentralized tactical decision-making. You must state the desired objective and the why behind it, but leave the how entirely up to subordinates on the ground.

Effective leadership under uncertainty requires mission command, combined with intellectual humility and sensitivity to scope.

During the invasion of Belgium in May 1940, German lieutenant Rudolf Witzig landed far off target during the assault on the fortress of Eben Emael. A sergeant took charge, knocked out the guns, and improvised successfully until Witzig arrived.

• Tell subordinates what goal to accomplish but not how to achieve it.
• Push decision-making power down the hierarchy to those encountering surprises.
• Draw a sharp line between deliberation and implementation.
• Adhere decisively to a chosen course of action until overriding reasons demand a change.

Scope Sensitivity

People often suffer from scope insensitivity, replacing difficult quantitative questions about scale with an easier emotional evaluation. In Daniel Kahneman's experiments, people were willing to pay the same amount of money to clean up a small region of lakes as they were to clean up all 250,000 lakes in Ontario.

Superforecasters successfully overcome scope insensitivity by factoring time frames and scale variations into their probabilistic estimates. When asked about the fall of the Assad regime, regular forecasters gave a 40 percent chance over three months and a 41 percent chance over six months, whereas superforecasters adjusted their estimates to 15 percent and 24 percent respectively.

Limits of Forecasting

Extreme rare events and unpredictable black swans pose severe limits on the scope of long-range forecasting. Nassim Taleb argues that historical outcomes are distributed like wealth rather than height, featuring fat tails that permit extreme events.

George Soros shorted the British pound in 1992 and earned an estimated $1.1 billion by selling sterling worth almost $10 billion under extreme market pressure.

10. What's Next and the Ten Commandments

Systematic scorekeeping and Bayesian question clustering can transform institutional decision-making and depolarize public policy debates.

Accuracy is frequently subordinated to political and tribal interests in a power-driven reality. Vladimir Lenin summarized this dynamic as kto, kogo—meaning who, whom—which describes politics as a ceaseless struggle for power. On September 18, 2014, 43 percent of Scots voted yes in the referendum, while 57 percent favored no, and 9 percent remained undecided two weeks prior, as analysts like Daniel Drezner confessed uncertainty.

Just as early attempts to introduce hospital scorekeeping and the End Result System faced resistance, evidence-based forecasting meets pushback. Boston doctor Ernest Amory Codman was suspended from a local medical society in 1915 for publishing a satirical cartoon about hospital efficiency statistics.

Complex macro-questions can be evaluated through Bayesian question clustering and pointillism by aggregating scorable sub-questions. IARPA evaluated North Korea by asking whether it would launch a multistage rocket between January 7, 2013, and September 1, 2013. Tom Friedman asked a provocative question on Iraq in 2002 before the 2003 US invasion. Adversarial collaboration helps test opposing views, such as when signatories warned Fed chairman Ben Bernanke of inflation in 2010.

Ten Commandments for Superforecasters

• Focus on questions where your hard work is likely to pay off.
• Don't waste time on easy clocklike questions or impenetrable cloud-like questions.
• Concentrate on questions in the Goldilocks zone of difficulty.
• Decompose the problem into its knowable and unknowable parts.
• Flush ignorance into the open.
• Expose and examine your assumptions.
• Dare to be wrong by making your best guesses.
• Conduct creative searches for comparison classes even for seemingly unique events.
• Pose the outside-view question: How often do things of this sort happen in situations of this sort?
• Translate vague-verbiage hunches into numeric probabilities.
• Conduct unflinching postmortems on both failures and successes.
• Master perspective taking, precision questioning, and constructive confrontation.
• Maintain constant mindfulness while following guidelines.

10 Key Ideas

1. The average expert is roughly as accurate at forecasting global events as a dart-throwing chimpanzee, proving that predictive accuracy requires rigorous measurement and probabilistic methods rather than intuitive expertise.

1. Human decision making relies on fast, automatic System 1 heuristics that generate overconfidence, requiring slow, deliberate System 2 analysis to overcome cognitive biases and confirmation traps.

1. Accurate forecasting requires precise numerical probability scales and Brier scoring to properly incentivize forecasters and track calibration over time.

1. Superior foresight begins with the statistical base rate of a broader class before making specific adjustments for the individual case.

1. Complex estimation problems are successfully solved by breaking them down into smaller sub-components using Fermi estimation and active open-mindedness.

1. Superforecasters act as intuitive Bayesians who update their beliefs in small, steady increments rather than swinging between stubborn inertia and erratic overreactions.

1. Maintaining a growth mindset allows top forecasters to treat their knowledge base as an unfinished program in perpetual beta.

1. Structured superteams eliminate groupthink and drive elite performance by fostering a pro-social culture of precision questioning and constructive confrontation.

1. Effective leadership under uncertainty requires mission command, which combines clear strategic goals with decentralized tactical decision-making pushed down the hierarchy.

1. Systematic scorekeeping and evidence-based forecasting can transform institutional decision-making and depolarize public policy debates, despite facing resistance from entrenched tribal interests.