Introduction
You live in an era where misinformation spreads faster than truth, leaving you to wonder how smart people fall for obvious lies. You might assume that false beliefs stem from personal stupidity, ideological bias, or a simple lack of education.
Cailin O'Connor and James Weatherall argue that ordinary, rational individuals routinely form false beliefs simply because of how information flows through social networks. Instead of blaming bad actors or flawed psychology, the authors use mathematical models to show how social structure itself drives collective delusion.
The following sections examine how science operates, how social networks trap groups in error, how industries manufacture doubt, and how media distortions amplify falsehoods. You will discover answers to several pressing questions:
- Why do intelligent people persistently ignore overwhelming scientific evidence?
- How do well-funded industries successfully manipulate public debates?
- Why do social networks sometimes make groups dumber rather than smarter?
- How can you recognize and resist the structural forces of propaganda?
1. What Is Truth and Scientific Consensus
Absolute certainty remains unattainable in empirical research. You must update your beliefs based on incoming evidence rather than waiting for unreachable proof.
While absolute certainty is unattainable, rational agents update their beliefs based on evidence, though industries weaponize philosophical skepticism and political interference to manufacture doubt.
The Problem of Induction and Manufacturing Doubt
Philosophical skepticism points out that all scientific evidence is inductive. Because past theories have been proven wrong, chemical and energy industries argue you can never be certain about environmental threats. Companies like DuPont used this pessimistic meta-induction to demand endless research and block regulations.
DuPont placed newspaper advertisements arguing there was no persuasive evidence linking chlorofluorocarbons to ozone depletion. Its CEO later wrote to the US Senate in March 1988 claiming no need for drastic emission reductions.
You should respond to uncertainty by using Bayes' rule to adjust your credence. Since you need to make choices that avoid catastrophic outcomes, proportion your beliefs to the available evidence.
Political Interference in Science
Real harm often occurs when political actors intentionally alter scientific consensus. The Reagan administration established the Acid Rain Peer Review Panel headed by William Nierenberg. Fred Singer inserted a dissenting chapter and White House officials revised the Executive Summary without the panel's consent.
Thomas Kuhn published The Structure of Scientific Revolutions in 1962, describing how normal science operates within a paradigm until a revolution changes the entire worldview. While cultural contexts and biases can influence scientific problems, a scientist's background views do not automatically invalidate rigorous empirical evidence.
Paul Gross and Norman Levitt published Higher Superstition in 1994, sparking the science wars by criticizing academic leftists and radical environmentalists like Jeremy Rifkin. The true threat to scientific authority comes from those who manipulate empirical knowledge for industrial interests, not from academic critiques of science.
2. Network Models of Belief and Social Influence
Scientific discoveries are rarely the work of lone geniuses. They emerge from the slow accumulation of knowledge across a community and network of interacting researchers.
Physician Jane Hightower linked heavy mercury poisoning to wealthy patients eating predatory fish like shark, swordfish, and tuna. This discovery built upon contributions from colleagues, EPA contacts, and news coverage.
Mathematical models of social networks reveal how full communication can backfire through the Zollman effect and how conformity bias traps groups in false consensus.
The Zollman Effect and Communication Dangers
Mathematical models of communication networks help explain how groups of agents share evidence and decide between competing actions under probabilistic conditions. Developed by Venkatesh Bala and Sanjeev Goyal, and adapted by Kevin Zollman, these computer simulations represent scientists as nodes connected by edges, updating their credences through Bayes rule based on their own outcomes and those of their neighbors.
Researchers use networks such as cycles, stars, wheels, and complete networks to simulate how agents choose between a known action A and a better unknown action B. The credence ranges between 0 and 1. An example credence of 0.7 indicates a 70 percent chance action B is better than action A. A credence threshold of 0.99 is required for a network to converge to a true belief.
Full communication within a scientific community can sometimes cause the entire group to adopt false beliefs due to misleading data, a phenomenon known as the Zollman effect. When scientists share all evidence across tightly connected networks, an unlucky string of misleading results can derail the entire group from the true theory. Limiting communication or maintaining diversity of beliefs protects pockets of researchers from bad data.
Gastroenterologist E. D. Palmer biopsied over a thousand patient stomachs and found no bacteria, which effectively halted research on the bacterial theory of ulcers for nearly thirty years until Warren and Marshall revived it. The Nobel Prize was awarded to Robin Warren and Barry Marshall in 2005. Böttcher and Letulle first argued bacteria caused ulcers in 1874. E. D. Palmer published his biopsy results showing no bacteria in stomachs in 1954, based on exactly 1000 patient stomachs biopsied. Qin Shi Huang reportedly died from taking mercury pills in 210 BCE. Bala and Goyal introduced their mathematical model in 1998.
Mutual mistrust among researchers who reach different conclusions can lead to stable scientific polarization where groups reject each other's evidence completely. Using Jeffrey's rule to update beliefs under uncertainty, scientists who assign lower trust to colleagues with opposing views will eventually stop updating based on their data. This creates polarized camps that are immune to corrective evidence.
The Lyme wars created deep divisions between mainstream rheumatologists advocating caution and Lyme-literate doctors treating chronic Lyme disease with long-term antibiotics, resulting in fierce public battles and threats of violence. Connecticut health officials took Polly Murray's case to Allen Steere in 1975. Twenty-five years later, Allen Steere began receiving death threats.
Even overwhelming empirical evidence can be rejected by a scientific community when it clashes with professional norms and status. When evidence is loud and clear, social factors such as professional pride and resistance to unhygienic implications can cause a community to reject life-saving practices for decades.
Ignaz Semmelweis discovered that requiring hand-washing with chlorinated solutions drastically reduced childbed fever deaths in Vienna, but upper-class physician colleagues rejected his findings because they were offended by the implication that their hands were unclean. Ignaz Semmelweis took a post in the Vienna General Hospital in 1846. The average percentage of patients killed by childbed fever in the first clinic was 10. Midwives lost 3 to 4 percentage of patients in the second clinic. Semmelweis had his breakthrough after a colleague died from an autopsy cut in 1847. Semmelweis lowered the death rate in Budapest to 1 percentage.
Conformity and Polarization
Rational individuals making inferences from the observable behavior of others can fall into information cascades and adopt collective false beliefs. When people observe the public actions of others and reasonably assume those actions are based on private information, they may override their own private evidence, leading an entire group to cascade into the wrong decision.
Investors observing others buy a particular stock will rationally infer those buyers possess special tips, causing subsequent investors to follow suit despite their own contrary evidence. Consider a hypothetical group of 50 in the stock-trading example. Forty-eight individuals hold secret information favoring Nissan. Only 2 individuals hold information favoring General Motors.
The human desire to conform and fit in with a social group can override truth-seeking behavior and trap communities in bad practices or false consensus. When social payoffs for agreeing with peers outweigh the benefits of performing optimal actions, agents will deny the evidence of their own senses or beliefs to avoid social friction.
In Solomon Asch's classic experiments, more than a third of study participants chose obviously incorrect lines to match unanimous confederates, demonstrating strong conformity bias. Psychologist Solomon Asch devised his conformity experiment in 1951. Audience polling on game shows expects 91 percentage of correct feedback. Asking a single friend on game shows yields 65 percentage of correct answers.
False or pseudoscientific beliefs persist easily when they carry no practical real-world costs and serve as social signals of group membership. When holding a mistaken belief has no negative consequences on personal well-being, social pressures and the desire to signal alignment with specific cultural, elite, or political groups will easily stabilize that false belief.
Beliefs regarding the special health benefits of organic foods or the dangers of irradiated foods act as social signals for new-age or elite groups because the practical costs of holding them are minor. Donald Trump was inaugurated in 2017, and the inauguration crowd size dispute began that same year.
3. Propaganda Strategies and Spurious Results
Faced with overwhelming evidence linking smoking to cancer, tobacco executives realized they could not prove smoking was safe. Instead, their goal was to create the appearance of scientific uncertainty by funding and promoting research that muddied the waters. This gave policy makers and smokers cover to ignore scientific consensus.
Industrial propagandists exploit low-powered studies, biased production, and selective sharing to manufacture uncertainty without directly denying scientific facts.
In December 1952, Reader's Digest published Cancer by the Carton, causing a public relations crisis and a dip in sales. This prompted tobacco executives to form the Tobacco Industry Research Committee and issue the Frank Statement advertisement in January 1954.
Mechanics of Spurious Results
Smaller studies with fewer data points have a higher chance of generating random, spurious results that point in the wrong direction. Propagandists can collect and publicize these isolated anomalies to mislead observers, whereas large-scale studies reliably reflect the true underlying effects.
Breaking a single study with sixty data points into six smaller studies of ten data points each allows a propagandist to cherry-pick three smaller studies that spuriously support the incorrect theory. By directly funding studies and selectively publishing only the trials that spuriously support the wrong action while burying unfavorable findings, a propagandist can cause policy makers to approach certainty that the wrong action is better.
Tobacco companies funded extensive research through the TIRC but hid their own internal studies that confirmed a direct link between smoking and cancer until they were revealed through legal action in the 1990s. By 1986, tobacco companies spent 130 million dollars on sponsored research, resulting in 2,600 published articles.
Propagandists can also manipulate public opinion without funding new research simply by selectively promoting independent studies that happen to support their interests. The Tobacco Institute published a newsletter called Tobacco and Health that highlighted studies suggesting no link between smoking and cancer, and quoted epidemiologist Richard Doll out of context to imply uncertainty.
Embedded Propagandists and Reputation
Industry can corrupt science without biasing individual researchers by financially boosting scientists whose existing methodologies happen to produce industry-friendly results. Over time, through academic replication and training, the population of scientists shifts to favor industry interests even though all researchers act in good faith.
The Electric Power Research Institute granted 486,000 dollars to a research project at the University of Rochester investigating methylmercury in fish, whose findings minimized the dangers compared to other studies. Similar funding included 10,000 dollars from the National Tuna Foundation and 5,000 dollars from the National Fisheries Institute.
Propagandists successfully weaponize the reputations of respected authorities to undermine public agendas and create the appearance of scientific controversy.
Fred Singer coauthored an article in 1991 using oceanographer Roger Revelle's name to argue that greenhouse warming was too uncertain for drastic action. Political commentators and opponents used this to undermine Al Gore's environmental agenda. Justin Lancaster settled his libel suit regarding Revelle's disputed coauthorship in 1994.
By consistently sharing biased data directly with other scientists, an embedded propagandist leverages network communication to counteract true evidence. Models show that a propagandist who introduces artificially lowered success rates for a superior action can successfully jam the consensus-forming process among communicating scientists.
4. Media Distortion and Interventions
Journalistic novelty bias and fairness norms amplify misinformation, but proper weighting standards and localizing political issues can help restore objective truth.
Online misinformation can rapidly cascade through social networks and mainstream media, leading real people to commit violent acts based on fabricated claims. Anonymous social media claims regarding leaked political emails were amplified by hyper-partisan blogs, translated into bizarre translation manuals by online sleuths, and eventually broadcast by major alternative media outlets, creating an unearned sense of legitimacy.
Edgar Maddison Welch drove six hours from Salisbury, North Carolina, to Washington, D.C., to Comet Ping Pong pizzeria armed with an AR-15 rifle to investigate a completely fictitious child sex ring allegedly run by Hillary Clinton. A specific tweet was shared 6,000 plus times within a week. Alex Jones had 2 million plus followers on YouTube and 730,000 followers on Twitter.
Journalistic focus on novel, surprising, or contrarian stories creates a systematic novelty bias that can distort your perception of evidence, even when reporters stick exclusively to real events. When a third party or journalist selectively shares only the most unlikely or surprising scientific results rather than the complete distribution of evidence, audiences can be led to converge on false beliefs despite the underlying scientific consensus. Journalistic norms of fairness that grant equal time to contrarian viewpoints can act as a powerful distorter, slowing down public convergence on true beliefs and empowering fringe actors. Providing equal platform proportions to opposing sides of a scientific or factual debate puts a strong finger on the scale in the wrong direction because evidence favoring the true belief naturally appears more often.
Foreign intelligence operations used targeted social media affinity groups and star network structures to sow social division and build trust before pushing users toward more extreme political positions. By creating community pages centered around shared non-political interests like animal love or civil rights, propagandists established trusted peer identities that allowed them to broadcast polarizing content directly to receptive segments of the population. 126 million US Facebook users were reached by Russian-produced political content. Accounts linked to the Russian government spent 100,000 plus dollars on political ads. Russian-linked accounts operated groups such as LGBT United, Blacktivist, and a Heart of Texas page advocating secession, alongside a Pokémon Go contest called Don't Shoot Us.
Regulatory Interventions and Proper Weighting
Combating misinformation creates an asymmetric arms race because any algorithmic measure or rule implemented to catch fake news quickly becomes a target for bad actors to circumvent. Following Goodhart's law, when a regulatory metric becomes a fixed target for measurement, interested parties will immediately adapt to game the system, forcing continuous and costly institutional adaptation. During the early 1900s period of the rat bounty in Hanoi under French colonial rule, the government offered a bounty for rat tails, which resulted in people cutting off tails and releasing rats to breed and produce more tails.
Scientific reporting standards should adopt proportional weighting models that allocate space to viewpoints based on their representation and citation frequency in reputable peer-reviewed databases.
Wikipedia's proper weighting standard ensures that fringe opinions do not receive unearned parity, aligning public descriptions directly with the true distribution of academic consensus. You can apply these principles when evaluating information sources:
- Include multiple scientific opinions or views in an article if they are expressed in peer-reviewed articles indexed by reputable databases.
- Allocate space to each opinion proportional to the number of published articles in high-impact journals that support the view.
- Scale the weight of each opinion according to the number of citations its supporting articles have received.
- Give greater weight to more recent articles and citations compared to older ones.
Shifting political discourse toward local issues reduces the distorting effects of conformity and abstract identity signaling by forcing policies to face immediate, observable real-world consequences. When policies have local ramifications, individual daily experience provides direct feedback that pushes back against industry spin and ideological polarization, making conformity less dominant. Statements made by highly connected individuals who break from the consensus of their own social network carry disproportionate persuasive power due to the social risk involved. When an insider or trusted political figure goes against the dominant tribal trends of their peers, listeners infer they must possess compelling evidence, making their message uniquely effective at shifting minority opinions. Arizona senator John McCain stating that climate change is real had a much stronger impact on the political right than the same statement made by Al Gore.
Unfiltered majority-rule democracy fails when citizens lack the expertise to evaluate complex facts and are systematically manipulated by well-funded propaganda, necessitating a practical form of well-ordered science. Allowing crucial public policies to be decided purely via a populist majority vote leads to a tyranny of ignorance. Society must reinvent institutions to ensure democratic governance remains genuinely responsive to objective facts.
10 Key Ideas
- Ordinary, rational individuals routinely form false beliefs simply because of how information flows through social networks rather than personal stupidity or a lack of education.
- Because absolute certainty remains unattainable in empirical research, rational agents must update their beliefs based on incoming evidence while industries weaponize philosophical skepticism and political interference to manufacture doubt.
- Mathematical models of communication networks help explain how groups of agents share evidence and decide between competing actions under probabilistic conditions.
- Full communication within a scientific community can sometimes cause the entire group to adopt false beliefs due to misleading data through a phenomenon known as the Zollman effect.
- Rational individuals making inferences from the observable behavior of others can fall into information cascades and adopt collective false beliefs when human desires to conform override truth-seeking behavior.
- Industrial propagandists exploit low-powered studies, biased production, and selective sharing to manufacture uncertainty without directly denying scientific facts.
- Smaller studies with fewer data points have a higher chance of generating random, spurious results that point in the wrong direction, which propagandists can collect and publicize to mislead observers.
- Journalistic focus on novel, surprising, or contrarian stories creates a systematic novelty bias that can distort your perception of evidence, even when reporters stick exclusively to real events.
- Combating misinformation creates an asymmetric arms race because any algorithmic measure or rule implemented to catch fake news quickly becomes a target for bad actors to circumvent following Goodhart's law.
- Wikipedia's proper weighting standard ensures that fringe opinions do not receive unearned parity, aligning public descriptions directly with the true distribution of academic consensus.