Introduction
You often struggle to measure intangible aspects of human culture such as fame, collective memory, and historical trends. Traditional historians and sociologists rely on limited archives and subjective interpretations. This makes large-scale cultural evolution difficult to study objectively.
Massive digital datasets and computational lenses allow you to quantify cultural evolution, track historical suppression, and predict societal trends.
Erez Aiden and Jean-Baptiste Michel introduce culturomics as a new quantitative approach that analyzes digitized texts to reveal patterns in human behavior. The book explores the lifecycle of fame, the mechanics of automated censorship, the decay of collective memory, and the predictable acceleration of innovation.
Uncharted answers several core questions about human history and culture:
- How can you track the rise and fall of famous individuals using word frequencies?
- What do linguistic shifts reveal about government censorship and collective forgetting?
- How fast do new technologies and words spread through society?
- What does the digital accumulation of human data mean for the future of psychohistory?
1. The Science of Fame and Cultural Lifecycle
You can study intangible concepts like fame by treating word frequencies as a measurable proxy and analyzing generations through epidemiological cohort methods.
Cleaning Big Data
Google assembled a shopping list of 130 million books using catalog information. These legacy card catalogs contained uncorrected mistakes like assigning the first publication date to every issue of a serial. Initial calculations suggested an office colleague had been popular in the sixteenth century due to scrambled metadata.
To turn poop-grade data into a reliable resource, the authors deployed algorithms called the Serial Killer to find serial publications and the Speed Dater to guess publication dates from text. They filtered the corpus down to 500 billion words across 7 languages, covering 5 centuries of history. This cleaned dataset contains 4 percent of all books ever published.
The Grand Unified Theory of Fame
Fame is difficult to define and measure. You can use word frequencies in books, termed phame, as a facsimile for fame. Just as the Wright brothers built a wooden chamber 6 feet long in 1903 as a wind tunnel to test airplane wings experimentally rather than relying on flawed theories, researchers must accept simplifications while accounting for random errors, false negatives, and sampling bias.
Scholar Carol Gilligan appears more phamous than actor Robert Redford or revolutionary Václav Havel because book writers discuss intellectuals more often than popular stars. Inspired by Kristian Andvord's epidemiological study of tuberculosis cohorts in Norway, the authors grouped famous individuals by birth year from 1800 to 1950. Andvord analyzed tuberculosis mortality by birth cohort to prove the disease was primarily affecting the young rather than growing older over time, showing that 1 percent of babies born in Trondheim between 1887 and 1891 died of tuberculosis before their first birthday.
By tracking the 50 most famous people for each year across 7,500 total severe fame victims, researchers eliminated generation-specific distortions like the invention of the internet. When averaged out, every cohort follows a Grand Unified Theory characterized by a long period of silence, an exponential debut, a peak at age 75, and a slow multi-century decline. Younger cohorts debut at younger ages, double faster, and have shorter half-lives.
- The 1800 cohort debuted at age 43 and had a fame half-life of 120 years.
- The 1950 cohort debuted at age 29 and had a fame half-life of 71 years.
Bill Clinton at his peak attained a word frequency in books comparable to common vegetables like lettuce and cucumber.
Infamy and Career Paths
Analyzing the top 25 members of 6 professions born between 1800 and 1920 shows distinct career paths. Actors achieve fame early in life but reach lower overall book prominence. Writers delay gratification for later fame, politicians wait until middle or old age before skyrocketing to supreme leadership, scientists achieve lower fame very late, artists fare poorly, and mathematicians almost never achieve public fame during their lifetimes. Carl Friedrich Gauss proved fundamental mathematical theorems at age 19, but public fame eluded mathematicians until after death.
Top politicians include 11 out of 25 who became US presidents and 9 out of 25 who became heads of state elsewhere. Acts of extreme evil and mass murder create infamy more efficiently than any other human endeavor. They dominate the historical record through intense media attention. Adolf Hitler, Joseph Stalin, and Benito Mussolini surged to numbers 1, 2, and 5 by 1950 after executing mass atrocities. Morally courageous figures like Abraham Lincoln never ranked above number 5.
Hitler's regime murdered 10 to 11 million innocent civilians and prisoners of war, Stalin killed 20 million Soviet citizens, and Mussolini's Ethiopian genocide resulted in 300,000 deaths. Writer Charles Dickens held the number one fame spot from 1880 to 1940 before Hitler surpassed him.
Finishing second in a monumental historical achievement results in a massive penalty in long-term fame compared to being first. Humans landed on the moon on July 21, 1969, after traveling 239,000 miles. Buzz Aldrin walked on the moon just 19 minutes and one one-hundredth of a second after Neil Armstrong, yet he is approximately 5 times less famous.
2. The Sound of Silence and Information Control
You can detect cultural suppression automatically by analyzing large datasets of books over time. A sudden drop in a person's name reveals systematic erasure.
Big data and ngram analysis can automatically detect censorship and cultural suppression by identifying sudden, severe drops in the frequency of a person's name or specific words over time.
By comparing fame during a regime like Nazi Germany to periods before and after, researchers calculate a suppression score. While normal historical data shows a bell curve centered around one, German language books under the Nazis shift dramatically left. More than 10 percent of tracked figures suffered a fivefold or greater drop in fame. Fewer than 1 percent of people have a score bigger than five in either direction in English ngram results.
Applying this automated detector to German books from 1933 to 1945 revealed massive drops for figures like Picasso, Walter Gropius, and Hermann Maas.
Automated Censorship Detection
Information control relies on severe penalties for dissenting voices. You observe parallel patterns of erasure across multiple authoritarian regimes in history.
The Nazi campaign against ideas began with systematic book burnings orchestrated by student groups using meticulous lists compiled by obscure librarian Wolfgang Herrmann.
The Deutsche Studentenschaft posted 12 theses against the un-German spirit and utilized blacklist catalogs compiled by Wolfgang Herrmann. This led to public book burnings across German university towns, destroying thousands of books and severely depleting public library collections. By 1938, the Essen public library removed 69 percent of books present prior to the regime.
On May 10, 1933, students led by Joseph Goebbels burned books by authors such as Karl Marx, F. Scott Fitzgerald, Albert Einstein, H. G. Wells, and Helen Keller.
Totalitarian Erasure and Blacklists
Ideological purges often extend to targeted art and political rivals. Adolf Hitler and Joseph Goebbels relied on Max Nordau's theories claiming avant-garde art was the product of mental dysfunctions. They established the Reichskulturkammer to purge German culture. This culminated in the Entartete Kunst exhibition in Munich, which displayed confiscated works in a chaotic, derisive manner to incite public scorn.
The Entartete Kunst exhibition featured works by Marc Chagall, Wassily Kandinsky, and Paul Klee, attracting over 2 million visitors in its first four months.
Joseph Stalin went beyond executing his political rivals to systematically erasing their records and contributions from the national memory of the Soviet Union. After forming a troika to undermine Leon Trotsky, Stalin turned on co-conspirators like Zinoviev and Kamenev, executing them during the Great Purge and condemning Trotsky in absentia. Ngrams show that the fame of these revolutionaries dropped precipitously and did not rebound until Mikhail Gorbachev's perestroika and glasnost reforms in the late 1980s.
Trotsky was assassinated in Mexico in 1940 with an ax blow to the head sent by Stalin's agent Ramón Mercader.
Post-World War II anti-communist anxiety in the United States led to congressional investigations and a studio-enforced blacklist that barred prominent filmmakers and screenwriters from working. The House Un-American Activities Committee investigated communist influence in Hollywood, leading to contempt citations for the Hollywood Ten who refused to testify. Forty-eight studio producers established a blacklist preventing them from finding work until they purged themselves of contempt.
Dalton Trumbo, a member of the Hollywood Ten, was unable to receive screen credits for over a decade until he was credited as the screenwriter for the film Exodus in 1960.
Severe government censorship and information suppression can successfully erase monumental historical events from contemporary public knowledge and digital discourse. Following the 1989 Tiananmen Square massacre, Chinese officials shut down newspapers and publishing houses, monitored digital media through the Great Firewall of China, and enforced official accounts. As a result, Chinese ngram records show only a transient blip compared to Western spikes, and young citizens often fail to recognize iconic symbols like the tank man.
Undergraduate students at Beijing University quizzed years later appeared not to recognize the iconic tank man image showing a protester standing in front of tanks.
3. Collective Memory and Accelerating Innovation
Society follows predictable curves of collective forgetting and learning, with the timescale of technological adoption shrinking rapidly over time.
Hermann Ebbinghaus pioneered the empirical study of memory by using nonsense syllables to discover the fundamental laws of learning and forgetting. To eliminate unwanted associations tied to meaningful words, Ebbinghaus memorized 2,300 synthetic consonant-vowel-consonant syllables.
Ebbinghaus used himself as the sole test subject for more than two years, following a strict schedule dictated by the ticking of a mechanical watch. By measuring his recitation times and recall rates under strict conditions, he mapped the learning curve and the forgetting curve, showing that nearly half of a list is forgotten within twenty minutes.
Mapping Collective Forgetting
By tracking yearly numerals in ngrams, researchers can map collective memory and prove that society possesses both a short-term and a long-term memory. Using numbers corresponding to years between 1800 and 2000 as a synthetic vocabulary avoids the unpredictable associations that plague words. The resulting charts reveal that collective forgetting follows a two-regime curve where interest drops rapidly in the first few decades and much more slowly thereafter.
The frequency of the year 1872 declined to half its peak value in 1896, a lag of twenty-four years, whereas 1973 dropped to half its peak value in only a decade. Researchers used 201 numbers between 1800 and 2000 to study collective forgetting.
Accelerating Collective Learning
Big ideas and technological inventions take decades to propagate through a society because they require supporting engineering, skills, economics, and infrastructure. Unlike breaking news that travels almost instantly, technological innovations involve complex ecosystems of manufacturing, use, and distribution. Analyzing a curated list of inventions through ngrams reveals the lengthy timescales required for a technology to reach cultural prominence.
Aviva compiled 147 big ideas and their birthdays. The first patent for the fax machine was awarded to Alexander Bain in 1843, and a commercial telefax service was established between Paris and Lyon in 1865, yet the technology soared to peak popularity in the 1980s. Blue jeans took 103 years to reach the quarter-impact mark.
The collective learning curve has been shrinking rapidly over time, indicating that human society is adopting new technologies faster and faster. By grouping 147 technological inventions by their date of invention and analyzing their ngram trajectories, researchers found that the time required to reach one-quarter cultural impact has steadily decreased by about 2.5 years every decade.
Early-nineteenth-century technologies took sixty-five years to reach the quarter-impact mark, whereas turn-of-the-century inventions took only twenty-six years. These inventions include the 1801 invention date of the Jacquard loom and the 1920 invention date of the theremin.
The accelerating pace of technological progress suggests that human history may approach a fundamental singularity beyond which human affairs cannot continue as before. Mathematician John von Neumann and futurist Ray Kurzweil observed that the exponential growth rates in technology, exemplified by computational power and Moore's law, point toward a future convergence where machine intelligence will surpass human capacity. Ray Kurzweil predicts that by the year 2045, an ordinary computer will possess more processing power than all of mankind's brains combined.
4. The Digital Lens and Psychohistory
The digitization of books, modern lifelogging, and cultural inertia create an interdisciplinary foundation for predicting human history through data.
Google and other entities are systematically digitizing millions of books to preserve human heritage. However, legislation such as the Copyright Term Extension Act of 1998 prevents online dissemination and digital research for almost all books published after 1923, leaving an information ecosystem hamstrung by legal restrictions. Copyright was extended by seventy years after the author's death. Sixty percent of traditional brick-and-mortar libraries have faced flat or declining budgets.
Organizations like the Internet Archive, HathiTrust, and Project Gutenberg strive to provide open access to books, but are heavily constrained regarding works from the last century. Out of 130 million books in existence, Google digitized 30 million books. Initially, 4 percent of all books ever published powered the Ngram Viewer. This grew to 6 percent or one in seventeen books covered by the Ngram Viewer after a 2012 upgrade. Tight budgets and lack of space are forcing public libraries to secretly destroy hundreds of thousands of physical books instead of giving them away. Because books contain anti-theft tracking equipment that is too expensive to remove, and because libraries lack space and funds amid declining budgets, they routinely resort to destroying physical volumes on a massive scale. Sometimes they check against Google's digitized copies before disposal. Volumes from the personal library of former British prime minister David Lloyd George were trashed during these disposal processes.
Unlike books, newspaper and manuscript digitization remains fragmented, creating massive blind spots in the historical record for uncelebrated figures. Famous figures receive star treatment while the 99 percent are largely ignored. The National Endowment for the Humanities funded digitization covering six million pages in Early American newspapers. Australia's Trove project digitized one hundred million newspaper articles. Edgar Allan Poe left behind 422 letters, but his 1844 balloon hoax published in the New York Sun cannot be found in its original newspaper digital format because the NEH only funded the Sun from 1859 to 1920.
The financial investment in understanding human history through data is minuscule compared to big science projects, representing a massive missed opportunity. The Large Hadron Collider cost $9 billion, the Human Genome Project cost $3 billion, while the annual budget of the National Endowment for the Humanities is about $150 million. Society routinely funds multibillion-dollar scientific endeavors, yet funding for preserving and digitizing cultural history remains comparatively tiny. This points to the need for large-scale collaborative big humanities projects.
The Digital Present and Psychohistory
The average modern person generates vastly more written and digital correspondence than historical icons, creating an intimate digital window into the quantified self. The average American adult sends 422 emails every other week. In 2010, two billion e-mailers sent 10 trillion e-mails excluding spam. An analysis of JB's email mailbox over the years showed his gradual linguistic shift from French to English and the unfolding of his love life.
Advances in wearable technology, high-resolution recording, and mind-machine interfaces are making comprehensive life logging and the recording of internal thoughts a near-future reality. The average person alive today produces one terabyte of data each year. Professor Deb Roy and Professor Rupal Patel founded the Human Speechome Project, outfitting their home with 11 high-resolution video cameras and 14 microphones to record the first three years of their son Dwayne Roy's life. The Roy family basement data center stored 300 gigabytes of information each day. The TotalRecall CPU grid system has a 1 petabyte storage capacity.
Digital records provide unprecedented power for law enforcement to solve crimes, but they can also permanently expose and torment innocent victims of abuse. During the Boston Marathon bombing in 2013, which caused 3 deaths and 14 amputation-requiring injuries among 500,000 spectators watching the race, the FBI used surveillance footage and crowdsourced civilian photographs to identify the suspects. Conversely, fifteen-year-old Rehtaeh Parsons faced relentless online bullying after explicit photos of her were circulated, leading to her suicide in April 2013.
The shift toward default data recording turns information preservation into a moral dilemma, fueling fears of corporate surveillance and dystopian government tracking. The 5 Point Café in Seattle banned Google Glass to preserve customer privacy. The startup Snapchat created messages that automatically delete after a specified time. Researchers from disparate fields are combining empirical quantitative analysis with critical humanistic inquiry to study cultural records, forming new paradigms known as digital humanities, corpus linguistics, and computational social science. Stanford scholar Franco Moretti uses digital books and computer science methods to study Shakespearean character networks, while literature professor Matthew Jockers analyzes pronoun distributions in nineteenth-century novels.
Inspired by Isaac Asimov's fictional concept of psychohistory and Auguste Comte's social physics, testing ngrams reveals that cultural terms in motion tend to stay in motion. This points toward a possible predictive science of history. An analysis of Ngram frequency charts over twenty-year periods demonstrated that consistent upward or downward trends continue steadily for decades or centuries afterward, establishing a principle of cultural inertia.
10 Key Ideas
- Massive digital datasets and computational lenses allow you to quantify cultural evolution, track historical suppression, and predict societal trends.
- When averaged out across generations, every famous cohort follows a Grand Unified Theory characterized by a long period of silence, an exponential debut, a peak at age 75, and a slow multi-century decline.
- Big data and ngram analysis can automatically detect censorship and cultural suppression by identifying sudden, severe drops in the frequency of a person's name or specific words over time.
- The Nazi campaign against ideas began with systematic book burnings orchestrated by student groups using meticulous lists compiled by obscure librarian Wolfgang Herrmann.
- Society follows predictable curves of collective forgetting and learning, with the timescale of technological adoption shrinking rapidly over time.
- The collective learning curve has been shrinking rapidly over time, indicating that human society is adopting new technologies faster and faster.
- The digitization of books, modern lifelogging, and cultural inertia create an interdisciplinary foundation for predicting human history through data.
- Tight budgets and lack of space are forcing public libraries to secretly destroy hundreds of thousands of physical books instead of giving them away.
- The financial investment in understanding human history through data is minuscule compared to big science projects, representing a massive missed opportunity.
- An analysis of Ngram frequency charts over twenty-year periods demonstrated that consistent upward or downward trends continue steadily for decades or centuries afterward, establishing a principle of cultural inertia.