Book summary “Everybody Lies”: summary

Introduction

You live in a world where people constantly hide their true thoughts, feelings, and intentions. When you ask people questions directly through traditional surveys, they lie to protect their social standing or conform to expectations. You cannot rely on conventional social science research because the act of questioning changes the answer. This systematic deceit leaves you with a distorted view of human nature, culture, and your own community.

Seth Stephens-Davidowitz argues that a revolutionary new tool solves this problem by capturing what people actually think rather than what they claim to think. Big Data acts as a digital truth serum, revealing our true thoughts, hidden prejudices, and actual behaviors by analyzing anonymous search queries. When you examine the massive digital exhaust left behind on search engines and online platforms, you bypass the filter of social desirability.

Everybody Lies explores how this digital lens exposes the hidden mechanics of human behavior across psychology, economics, and sociology. You will discover how anonymous data shatters long-held assumptions about desire, prejudice, geography, and human decision-making. The book addresses several fundamental questions about the nature of modern information.

• What do anonymous search queries reveal about hidden human desires and prejudices?
• How do digital footprints outperform traditional psychological theories and surveys?
• Where do geographic patterns expose the deep environmental roots of success and failure?
• Why do standard economic and political models fail to predict human choices accurately?
• How can you leverage Big Data while avoiding the analytical traps of false correlations?

1. Your Faulty Gut

At its core, data science is about spotting patterns and predicting how one variable affects another. You apply this exact methodology naturally in everyday life. The author's eighty-eight-year-old grandmother successfully evaluated relationship qualities based on decades of observed data points.

Good data science mirrors intuitive human pattern recognition, but computers are necessary because human intuition fails to process massive scale and suffers from cognitive biases.

The Limitations of Human Intuition

While your gut can provide general insights, it suffers from cognitive traps such as overweighting personal experience. You also tend to overestimate the prevalence of dramatic stories. Computers and large datasets are required to correct these predictable errors.

Researchers from Columbia University and Microsoft utilized anonymous Bing search engine queries to successfully predict pancreatic cancer symptoms with high precision. This approach isolates signals that escape human observation.

• Pancreatic cancer has a low five-year survival rate of 3 percent.
• The algorithm identified 5 to 15 percent of pancreatic cancer cases with almost no false positives.
• Honolulu recorded 40 percent fewer depression searches than Chicago during winter months.

Socioeconomic Realities in Professional Sports

Intuition often misleads you when evaluating the roots of success and failure. Statistical analysis of county-level births, parental marital status, and players' first names reveals surprising truths about elite athletes. Contrary to conventional wisdom, professional basketball players are more likely to come from middle-class, stable backgrounds than from poverty. Better socioeconomic status provides an advantage in reaching the NBA due to nutrition, height gains, and developed social skills.

Basketball prospect Doug Wrenn possessed elite athletic talent but failed to develop the temperament required for a professional career. He contrasted sharply with Michael Jordan, who benefited from a stable, middle-class upbringing.

Several stark demographic markers define this reality:

• Black men are 40 times more likely than white men to reach the NBA.
• About 60 percent of all African-Americans born in the 1980s had unmarried parents.
• The average height of an NBA player reaches 6'7", compared to 5'9" for an average American man.

2. Was Freud Right?

Sigmund Freud claimed that dreams and slips of the tongue expose hidden desires. You might assume that modern digital records validate these classic psychoanalytic theories.

Big Data testing proves that Sigmund Freud's theories regarding phallic dream symbols and slips of the tongue are false, while revealing unexpected echoes of Oedipal concepts in pornography searches.

Testing Psychoanalytic Theory

You can test dream symbols by analyzing recorded data. Using a large dataset of recorded dreams from the app Shadow, researchers tracked phallus-shaped foods. Bananas and cucumbers appear in dreams strictly in proportion to how frequently people consume them in waking life. Phallus-shaped foods do not appear in dreams with any higher frequency than can be explained by their general consumption popularity.

Speech errors also fail to support psychoanalytic claims. Microsoft researchers collected a dataset of 40,000 typing errors to see if human typos reveal repressed sexual desires. An automated error bot made accidental sexual misspellings like sexshell and lipsdick at the exact same rate as real humans making typos. Sexually oriented slips occur no more often than expected by chance.

Pornography search data reveals unexpected results regarding other psychoanalytic concepts. Analysis of search queries and video views on mainstream platforms like PornHub indicates high explicit interest in incestuous scenarios, aligning with Freud's emphasis on the psychological importance of early childhood and mothers.

Several clear metrics illustrate this unexpected pattern:

• 16 of the top 100 searches by men on PornHub involve incest-themed videos, with mother-and-son scenarios being the plurality.
• 9 of the top 100 searches by women on PornHub look for incest-themed videos.

The Four Powers of Big Data

You can understand the revolution in social science by examining how digital archives operate. Big Data derives its revolutionary power from four unique capabilities that distinguish it from traditional research methods.

The unique powers of Big Data include:

• offering completely new types of data sources;
• providing honest and uninhibited disclosures due to digital anonymity acting as a digital truth serum;
• allowing researchers to zoom in on tiny population subsets;
• enabling rapid controlled experiments for causality.

3. Data Reimagined

Unconventional digital footprints, digitized texts, visual data sources, and prediction-oriented metrics track real-world economic and cultural shifts faster than traditional research methods.

You can understand the revolution in social science by examining how digital archives operate. Google outperformed early search engine competitors not by accumulating more raw data, but by collecting a fundamentally superior type of data. Early search engines simply counted keyword frequencies, making them easy to manipulate with hidden text. Google revolutionized search by analyzing hyperlinks between websites as a crowdsourced endorsement of relevance.

Typing Bill Clinton into late-1990s search engines brought up random joke sites. Google used inbound link analysis to direct users to the official White House website.

Tracking Economic Indicators

Unconventional data sources like internet search queries can track real-world economic indicators much faster than official government surveys. While government unemployment statistics take roughly two billion milliseconds to compile and release, researchers can look at high-frequency digital traces such as search engine queries to monitor economic trends in near real-time.

Seth Stephens-Davidowitz entered US unemployment rates into Google Correlate and found strong correlations with diversion searches like Slutload and Spider Solitaire, reflecting bored unemployed people spending time at home.

In predictive data science, knowing that a correlation works is far more valuable and practically sufficient than understanding the underlying causal mechanism. When building models to forecast future events, analysts do not strictly need to explain why certain variables correlate with outcomes. They simply need the statistical relationship to hold true.

Jeff Seder used portable ultrasounds to discover that the size of a racehorse's left ventricle predicts racing success. The future Triple Crown winner American Pharoah possessed a massive left ventricle size at the 99.61 percentile as a one-year-old. Walmart discovered that strawberry Pop-Tarts sales increase sevenfold before hurricanes, without fully understanding the underlying biological or psychological reasons.

Words, Pictures, and Predictions as Data

Digitized texts and historical books allow researchers to quantify cultural shifts and national identity changes through linguistic analysis. By tabulating word choices and grammatical patterns across massive corpuses of digitized books over centuries, researchers can track how societies evolve from fragmented states into unified nations.

Erez Aiden and Jean-Baptiste Michel used Google Ngrams to analyze historical books from 1800 to 2010. They discovered that the linguistic transition of describing the United States from a plural noun to a singular noun happened much more gradually after the Civil War than historians believed.

Digital recordings of speech during social interactions can reveal hidden romantic interest and compatibility better than traditional demographic surveys. By analyzing word choice, pitch modulation, pronoun usage, and laughter on first dates, researchers can predict whether participants will report a mutual connection and desire a second date. Scientists studying speed daters found that women signal interest by using self-referential words like I and self-marking phrases like Ya know?, while men signal interest by laughing at jokes and keeping pitch ranges narrow.

Political bias in mainstream news media is driven primarily by market demand and capitalism rather than top-down ideological conspiracies by owners. Economists Matt Gentzkow and Jesse Shapiro analyzed congressional speech patterns and 433 digitized newspapers to measure political slant. They found that publications adopt the ideological leaning of their local readership to maximize profits.

The New York Times Company owned both the liberal New York Times in heavily Democratic New York City and the conservative Spartanburg Herald-Journal in Republican Spartanburg, South Carolina.

Non-traditional visual data sources can accurately measure macroeconomic indicators in developing regions. Where official government economic statistics are missing or unreliable, alternative visual inputs provide timely insights into GDP growth and inflation. U.S. Air Force satellites circle the earth 14 times per day to photograph night light levels. Joseph Reisinger founded the company Premise, which employs workers in developing countries to photograph supermarket fruit bins and gas station lines to track inflation and economic health before official data is published.

4. The Digital Truth Serum

Google searches serve as a digital truth serum because digital anonymity removes social desirability bias, exposing hidden demographics, explicit prejudices, and unspoken personal crises.

People routinely lie on anonymous surveys due to social desirability bias and the desire to make a good impression. In contrast, online searches provide immediate personal utility and lack a human interviewer, making you reveal your true, uncensored thoughts, worries, and private behaviors.

University of Maryland graduates claimed in surveys that none of them graduated with a GPA under 2.5 when the real figure was 11 percent, and 44 percent claimed they donated to the university when only 28 percent actually did.

Uncovering Hidden Populations and Biases

Big Data analysis of porn searches and Facebook data indicates that about five percent of American men are gay, a much higher figure than traditional surveys suggest. Because social stigma drives gay men in intolerant states into the closet and leads them to lie on surveys, researchers can use male porn searches as a more objective proxy. Approximately five percent of male porn searches are for gay porn across both tolerant and intolerant states, revealing that millions of men remain closeted.

In Mississippi, only two in one thousand male high school students are openly gay on Facebook, yet roughly 4.8 percent of male porn searches in the state are for gay porn. By contrast, 5.2 percent of male porn searches in Rhode Island are for gay porn.

Pornography search data reveals that a significant minority of women have fantasies involving violence or nonconsensual sex, which they do not discuss in real life. Fully 25 percent of female searches for straight porn emphasize the pain or humiliation of the woman, and five percent search for nonconsensual sex terms.

Explicit racist Google searches correlate more strongly with real-world discrimination and political outcomes than implicit-association tests do. Nate Silver found that the geographic variable correlating most strongest with support for Donald Trump in the 2016 Republican primary was the map of racist Google searches. Over 7 million American searches every year include the word nigger, making it 17 times more common to search for nigger jokes than other ethnic slurs combined.

Appealing for tolerance during a crisis can inadvertently inflame public rage rather than calm it, as demonstrated by minute-by-minute search data. During President Obama's 2015 speech advocating tolerance after the San Bernardino shooting, searches for the phrase kill Muslims tripled, accompanied by a 60 percent increase in negative searches about Syrian refugees and a 35 percent drop in searches asking how to help Syrian refugees.

Parental inquiries into children's intelligence reveal a persistent gender bias, with parents disproportionately searching whether their sons are gifted or geniuses. Parents are two and a half times more likely to ask if their son is gifted than if their daughter is gifted, despite 9 percent more likely chances for girls to be in gifted programs in American schools.

Social media platforms like Facebook function as brag-to-my-friends serum rather than truth serum, causing you to heavily curate your life to look successful and happy. Because social media profiles are tied to your real identity and social circle, you have strong incentives to broadcast highbrow culture, luxury vacations, and perfect marriages. You hide messy realities, financial struggles, and private vices. While highbrow magazine The Atlantic and sensational tabloid National Enquirer have similar real-world circulations and Google search volumes, The Atlantic enjoys nearly thirty times more likes and discussions on Facebook. There are 27 Atlantic likes or mentions for every 1 National Enquirer mention.

Economic studies show that the internet is remarkably desegregated politically, exposing you to opposing views more frequently than offline interactions do. Economists Matt Gentzkow and Jesse Shapiro calculated that the probability that two people visiting the same news website have opposing political views is about 45.2 percent. This rate is higher than for coworkers at 41.6 percent, family members at 37 percent, or friends at 34.7 percent.

Official government reports can falsely indicate a drop in severe societal problems like child abuse during recessions because official reporting systems break down. During the Great Recession, official child abuse reporting dropped. However, Google searches by children stating their parents hit them surged in close correlation with the rising unemployment rate. This trend reflects a 3 percent increase in search rates for child abuse or neglect for every percentage point increase in unemployment.

When legal abortion access is restricted by state laws, women turn to the internet in large numbers to research self-induced and off-the-books abortion methods. In 2015, there were over 700,000 Google searches looking into self-induced abortions in the United States. Women also performed 160,000 searches for getting abortion pills through unofficial channels and 4,000 searches looking for directions on coat hanger abortions, while states with the fewest abortion clinics saw 54 percent fewer legal abortions.

5. Big Psych and Geography

Granular geographic data reveals how local environments shape economic mobility, political imprinting, tax compliance, fame, and life expectancy.

Your favorite sports team as an adult is largely determined by whether that team won a championship during your early childhood. Analyzing Facebook likes across various baseball teams shows that the popularity of a team peaks significantly among men who were around seven or eight years old when the team won a World Series. Winning when you are nineteen or twenty is about one-eighth as important as winning when you are eight. Seth Stephens-Davidowitz found a surge in New York Mets fans among men born in 1962 and 1978, corresponding to the team's World Series wins in 1969 and 1986.

Political preferences are largely imprinted during a crucial period in late adolescence and early adulthood based on the popularity of the current president. Researchers Yair Ghitza and Andrew Gelman analyzed over 300,000 survey observations across sixty years and found that the single most important year for developing political views is age eighteen. The impressionable window spans ages fourteen to twenty-four. Americans coming of age under popular Republican Dwight D. Eisenhower tilted Republican for life. Baby boomers coming of age under popular Democrats John F. Kennedy and Lyndon B. Johnson tilted liberal their entire lives. This imprinting produced a 10 percentage point lifetime boost for Republicans among Americans born in 1941 and a 7 percentage point advantage for Democrats among Americans born in 1952.

Imprinting and Economic Mobility

National averages mask enormous local variations in economic mobility, proving that some parts of a country are true lands of opportunity while others trap people in poverty. Analyzing 1.2 billion IRS tax records, Raj Chetty and his research team demonstrated that the chance of a poor child reaching the top 20 percent of income varies dramatically by geography. Local education spending, religion, crime, and the presence of single mothers influence this chance. Poor children in San Jose, CA, had a 12.9 percent chance of reaching the top income quintile, compared to just 4.4 percent in Charlotte, NC. The United States average chance sits at 7.5 percent.

While wealthy people experience high life expectancies regardless of location, the life expectancy of impoverished individuals varies drastically depending on the city they live in. Chetty's team discovered that the presence of a high concentration of rich people in a city is the primary predictor of longer life spans for poor residents, likely driven by contagious healthy behaviors. American women in the top 1 percent of income live on average fifteen years longer than those in the bottom 1 percent for men, and ten years longer for women, with poor people gaining an extra five years of life expectancy by living in the right geographic location.

Tax fraud is often driven more by informational access and social contagion than by an inherent lack of honesty. Analyzing self-employed taxpayers claiming the Earned Income Tax Credit, researchers found that reporting the exact optimal income of $9,000 to receive a $1,377 government check clustered heavily in certain regions like Miami. This behavior spreads outward like a virus through neighbors and tax professionals. In Miami, 30 percent of self-employed taxpayers with one child reported the exact magic taxable income of $9,000, compared to just 2 percent in Philadelphia.

Geography of Success and Predictive Modeling

The likelihood of a person achieving notable success or fame depends heavily on early exposure to innovative environments like college towns and urban centers. By downloading Wikipedia and merging it with birth statistics, Stephens-Davidowitz found that notable individuals are disproportionately born in counties with major universities, dense urban areas with successful role models, and higher proportions of immigrants. Black children born in Tuskegee, Alabama, home of Tuskegee University, had the same probability of becoming notable outside of sports as white children born in top majority-white college towns. Overall, the probability of an American-born baby boomer earning a Wikipedia entry among 150,000 notable Americans was 1 in 2,058. This breaks down as:

• 1 in 1,209 for a baby boomer born in California;
• 1 in 4,496 for a baby boomer born in West Virginia;
• 1 in 852 for a baby boomer born in Macon County, Alabama.

Exposure to violent movies significantly decreases immediate local crime rates by keeping potentially violent young men off the streets and away from alcohol. Economists Gordon Dahl and Stefano DellaVigna merged FBI crime data, box-office numbers from 1995 to 2004, and movie violence ratings to show that crime drops significantly on weekends when a prominent violent movie is released. Aggressive individuals choose theaters over bars and clubs instead. When violent films like Hannibal or Dawn of the Dead played in theaters, alcohol-related crimes plummeted during late-night hours because young men were safely seated in unarmed environments.

Accurate long-term predictions about human behavior and performance can be achieved by finding historical doppelgangers who share exact developmental trajectories. Nate Silver revolutionized sports forecasting with sabermetrics using the PECOTA system by searching massive databases of 18,000 Major League Baseball players for the twenty players whose physical and statistical profile most closely matched a player's early career. When Boston Red Sox slugger David Ortiz hit a severe slump in 2009 with a .270 batting average and experts declared him finished, his doppelgangers like Jorge Posada and Jim Thome showed the way. Power hitters often experience a late-twenties dip followed by a powerful career resurgence, leading to Ortiz being voted World Series MVP in 2013 at age 37.

6. All the World's a Lab

Randomized controlled experiments, digital A/B testing, natural experiments, and regression discontinuity studies provide rigorous causal insights across economics, education, and business.

Methodology of Causality

Randomized, controlled experiments are the gold standard for establishing causal relationships rather than mere correlations. By randomly dividing participants into treatment and control groups, researchers eliminate reverse causation and omitted-variable bias. You can then trust that observed differences are truly causal effects.

Economist Esther Duflo studied education in rural India by randomly dividing schools into groups where teachers received extra pay for showing up to work. In some schools in rural India, 40 percent of teachers were absent on a given day. Teachers received 50 rupees of extra daily pay for showing up to work, or about $1.15.

The digital world revolutionizes randomized experiments by making them vastly cheaper, faster, and easier to conduct at any time. This represents the fourth power of Big Data. Unlike offline experiments that require significant resources, digital experiments deploy via simple code lines. They automatically measure clicks and user behavior without explicit participant consent.

Barack Obama's presidential campaign used Dan Siroker's Optimizely to A/B test website layouts, pictures, and buttons to maximize engagement. The winning website combination increased sign-ups by an estimated 40 percent, which netted the Obama campaign $60 million in additional funding. Google engineers ran 7,000 A/B tests in 2011.

Natural Experiments and Educational Impact

When intentional experiments are unethical or impossible, economists utilize natural experiments driven by the arbitrary randomness of life. By comparing outcomes across situations where chance dictates the result, researchers can isolate causal effects without violating institutional review boards.

Economists Benjamin F. Jones and Benjamin A. Olken studied leaders who survived or died in assassination attempts to determine the impact of leadership on national trajectories. Out of 43 total American presidents, 16 have been victims of serious assassination attempts, and 4 have been killed.

Natural experiments also emerge from unexpected events like sports playoffs. Because companies produce ads before knowing which teams will qualify, game outcomes create a natural experiment where specific regional audiences are randomly exposed to higher ad volume.

Researchers Hal Varian, Michael D. Smith, and the author analyzed Super Bowl broadcasts from 2004 to 2013 to measure movie ticket sales spikes in qualifying team cities:

• Boston recorded a 60.3 audience share for the 2012 Super Bowl;
• Baltimore recorded a 50.2 audience share for the 2012 Super Bowl;
• a movie ad slot cost an average of $3 million;
• increased ticket sales resulted in an $8.3 million return yielding a 2.8-to-1 return on investment;
• economists Wesley R. Hartmann and Daniel Klapper found a 2.5-to-1 return on investment for beer and soft drink ads.

Regression discontinuity studies reveal that attending an elite high school may have a negligible causal effect on a student's long-term academic success. By comparing students who barely missed an admission cutoff with those who barely made it, researchers can isolate the school's true impact from innate talent and drive. A recent study used 27 million reviews and 300,000 students and parents to rank public high schools. Every November, 27,000 New York youngsters sit for the Stuyvesant admission exam, and only 5 percent of test-takers get in. Economists Atila Abdulkadiroğlu, Joshua Angrist, and Parag Pathak studied New York pupils on both sides of the Stuyvesant High School cutoff and found zero impact on AP scores, SAT scores, or college rankings.

7. Limitations of Big Data

Massive digital archives invite endless calculation, but enormous power requires strict methodological discipline. When you analyze a dataset with thousands of variables, random coincidence will produce misleading patterns.

Careless data analysis risks falling victim to the curse of dimensionality and the overemphasis on easily measurable metrics at the expense of human realities.

The Curse of Dimensionality

The curse of dimensionality occurs when testing too many variables against too few observations leads to finding false, random correlations. When massive datasets provide an exponential number of variables to analyze, random chance guarantees that some variable will appear statistically significant, mirroring a lucky coin flip.

Computer scientists from Indiana University and Manchester University attempted to predict the stock market using Twitter sentiment analysis across numerous emotions and time delays. They evaluated 504 trading days over two years. If you flip 1,000 hypothetical coins, a random coin will occasionally appear to predict market rises 70.3 percent of the time purely by chance. Robust research requires a strict out-of-sample test to ensure that patterns hold beyond the initial dataset.

The Danger of Optimizing Measurables

An uncritical reliance on Big Data can cause decision-makers to fixate on easily measurable numbers while losing sight of broader, more important human realities. Because many critical qualities like critical thinking, curiosity, or user experience cannot be easily captured in simple metrics, optimization often leads to perverse incentives or distorted outcomes.

Yale marketing professor Zoë Chance became dangerously obsessed with daily step counts on a pedometer, walking continuously until she fell and discarded the device. She recorded 20,000 steps in a single twenty-four-hour period before abandoning the gadget. Similarly, Nate Silver estimated that the Oakland A's gave up 8 to 10 wins per year due to poor defense while chasing traditional sabermetric stats. Small data and human judgment remain essential counterweights to prevent algorithmic blindness.

8. Empowered Corporations and Governments

While Big Data empowers corporations to extract maximum consumer profit and predict behavior, attempting to use aggregate search data to predict individual criminal actions mathematically fails due to false positives.

Corporate Exploitation and Linguistic Signals

Corporations now use advanced digital signals to evaluate your reliability and squeeze maximum profit from your weaknesses. Economists analyzed data from the peer-to-peer lending site Prosper to uncover linguistic predictors of loan repayment. Overall, 13 percent of borrowers defaulted on their loan.

Corporate Exploitation and Linguistic Signals

Certain words signal whether you will pay back money. Financial sophistication terms like after-tax or debt-free correlate with repayment. Making heavy promises signals a high risk of default. Invoking God makes a borrower 2.2 times more likely to default. Asking for mercy because of a sick family member indicates high default risk.

Cambridge University and Microsoft tested 58,000 U.S. Facebook users and found that seemingly harmless online preferences correlate with personal traits. Liking Mozart correlates with higher IQ, while liking Harley-Davidson correlates with lower IQ. Algorithms can use these proxy variables for subtle employment discrimination.

Companies also use data science and doppelganger searches to estimate your exact financial pain point. Harrah's utilizes the Big Data warehouse firm Terabyte to track customer behavior and offer free steak dinners when regulars approach their loss limit of around $3,000.

The Limits of Predictive Policing

Big Data gives consumers powerful leverage through comparison sites and reviews. Economist Michael Luca compared Yelp reviews to sales data in Washington state and showed that losing a single star causes a 5 to 9 percent drop in restaurant revenues. Insurance comparison shopping generates 1 billion dollars per year in total consumer savings.

Governments attempt to use search data for security, but predicting individual crimes mathematically fails. Troubling thoughts are extremely common while actual violent crimes are rare, producing an overwhelming number of false positives.

Consider the massive volume of dark queries:

• there are 3.5 million Google searches in the U.S. related to suicide monthly, numbering 4,000 fewer than monthly suicides in the United States;
• Americans made 12,000 searches for kill Muslims in 2015 alongside 12 reported murders of Muslims as hate crimes;
• users submitted 6,000 searches for how to kill your girlfriend in 2014 alongside 400 girlfriend murders.

Adriana Donato was murdered by an ex-boyfriend who had searched for how to commit murder. This case illustrates the tragic limits of predicting individual violence despite clear aggregate search correlations.

9. Introduction to Big Data

Traditional surveys and polls are unreliable because people systematically lie or exaggerate about their behaviors and beliefs. When you answer questionnaires, you often present your desired self rather than your actual actions.

Traditional surveys fail because people lie about sensitive behaviors, whereas Google search data provides an honest, massive dataset reflecting actual human thoughts.

General Social Survey respondents report impossible numbers of condom use and sexual frequency that contradict actual retail sales tracked by Nielsen. Women report an average heterosexual sex frequency of 55 times per year and a condom usage frequency of 16 percent. They claim 1.1 billion condoms used annually while men claim 1.6 billion. Nielsen retail data shows a total of only 600 million condoms actually sold annually.

The Unreliability of Surveys

Human beings generate 2.5 million trillion bytes of information on an average day in the early 21st century. Google search data is the most important dataset ever collected on the human psyche because people confide in the search engine without social desirability bias. You use search bars to ask private questions, express hidden desires, and admit taboo thoughts.

Queries typed into Google include confessions like:

• I hate my boss;
• I am drunk;
• My dad hit me.

Uncovering Taboos and Cultural Differences

Anonymous search data reveals a vast amount of hidden racial animus in America that traditional political surveys entirely missed. On Obama's first election night, about one in every hundred Google searches containing Obama also included kkk or racial slurs, and Stormfront sign-ups spiked tenfold.

Big Data uncovers unexpected cultural differences and highly specific sexual and bodily insecurities across different global populations. In India, searches and pornographic requests for female breastfeeding of adult men are exponentially higher than anywhere else in the world, while women's top bodily insecurity on Google focuses on vaginal odor.

Data quality matters more than mere volume. Jerry Friedman recalled a 1977 Dallas conference on Large and Complex Datasets where statisticians bragged about having gigabytes or terabytes of data rather than discussing meaningful insights. An early distinguished statistician proudly claimed to have accumulated 5 gigabytes of data.

Conclusion

Traditional social science relies on what people say, which distorts reality through social desirability bias and cognitive limitations. You can overcome these blind spots by treating the digital exhaust of humanity as a vast behavioral mirror. Big Data acts as a digital truth serum, revealing our true thoughts, hidden prejudices, and actual behaviors by analyzing anonymous search queries rather than relying on flawed traditional surveys. When you combine digital footprints with causal experiments and machine learning, you transform guesswork into precise measurement.

You must apply specific rules to navigate this new era of information effectively:

1. Look at what people actually do rather than what they claim they do when you study human behavior.
2. Exploit digital exhaust such as search queries and clicks to uncover hidden preferences that surveys conceal.
3. Avoid relying on your gut feelings when data scale and complexity exceed your cognitive capacity.
4. Distinguish correlation from causation by demanding rigorous experimental designs before you alter policies or strategies.
5. Account for the curse of dimensionality and the risk of false positives when you analyze massive datasets.
6. Guard against the temptation to optimize only for easily measurable metrics while ignoring deeper human realities.
7. Respect individual privacy and data limitations while you leverage aggregate insights to understand social trends.

10 Key Ideas

1. Computers and large datasets are required to correct predictable human cognitive errors like overweighting personal experience and processing massive scale.
2. Big Data testing proves that Sigmund Freud's theories regarding phallic dream symbols and slips of the tongue are false, while revealing unexpected echoes of Oedipal concepts in pornography searches.
3. Unconventional digital footprints, digitized texts, visual data sources, and prediction-oriented metrics track real-world economic and cultural shifts faster than traditional research methods.
4. Google searches serve as a digital truth serum because digital anonymity removes social desirability bias, exposing hidden demographics, explicit prejudices, and unspoken personal crises.
5. Granular geographic data reveals how local environments shape economic mobility, political imprinting, tax compliance, fame, and life expectancy.
6. Randomized controlled experiments, digital A/B testing, natural experiments, and regression discontinuity studies provide rigorous causal insights across economics, education, and business.
7. Careless data analysis risks falling victim to the curse of dimensionality and the overemphasis on easily measurable metrics at the expense of human realities.
8. While Big Data empowers corporations to extract maximum consumer profit and predict behavior, attempting to use aggregate search data to predict individual criminal actions mathematically fails due to false positives.
9. Traditional surveys fail because people lie about sensitive behaviors, whereas Google search data provides an honest, massive dataset reflecting actual human thoughts.
10. Big Data acts as a digital truth serum, revealing our true thoughts, hidden prejudices, and actual behaviors by analyzing anonymous search queries rather than relying on flawed traditional surveys.