Introduction
You face a world flooded by an unstoppable surge of digital information. Traditional database tools fail when handling massive volumes of unstructured records generated at unprecedented speeds.
Saswat Sarangi demonstrates how this digital explosion redefines modern business, governance, and daily life. You will discover how organizations harness complex analytics and artificial intelligence to extract strategic value from chaotic datasets.
The book traces a clear path from fundamental data characteristics to advanced predictive applications and severe governance risks. You will learn to navigate the delicate balance between commercial advantage and ethical responsibility.
- What defines the core dimensions of massive datasets?
- How do analytical models turn raw figures into actionable insights?
- Which strategies protect privacy and prevent algorithmic failures?
1. Characteristics and Growth of Big Data
Defining Big Data and the 3Vs
Big Data refers to datasets so massive and fast-moving that they exceed the processing capacity of traditional database software tools. You encounter data volumes ranging from a few dozen terabytes to multiple petabytes. A June 2011 report formally defined these extreme data challenges.
Big Data encompasses massive datasets defined by the 3Vs of volume, velocity, and variety, driving exponential global growth and fueling artificial intelligence developments.
The foundational framework is the 3Vs model. A 2001 research report by META Group defined these dimensions to explain emerging growth challenges:
- Volume involves observing massive quantities of information without sampling.
- Velocity demands real-time data availability and processing.
- Variety requires drawing from structured and unstructured sources like text, images, audio, and video.
Data Storage and Technology Drivers
Digital storage measurements have evolved rapidly from early megabytes on floppy disks to massive modern units. Computer data storage units scale across multiple tiers:
- Bit and Byte
- Kilobyte and Megabyte
- Gigabyte and Terabyte
- Petabyte and Exabyte
- Zettabyte and Yottabyte
A 3-1/2 inch floppy disk holds 1.44 Megabytes, while modern computers commonly use 500-Gigabyte hard drives. A 64 GB pen drive capacity handles everyday personal tasks. One terabyte holds about 3.6 million 300 Kilobyte images.
Global data volumes expand at roughly 50 percent annually, reaching multi-quintillion byte scales daily. Domo's Data Never Sleeps report highlighted that 90 percent of all data today was created in the last two years. You generate 2.5 quintillion bytes of data per day. By June 2017, 77 percent of U.S. adults owned a smartphone.
The cost of digital storage dropped exponentially over the decades. World storage capacity grew from 2 billion gigabytes in the mid-1980s to 300 billion gigabytes by late 2007. The cost for 1 GB of storage fell from $438,000 in 1980 to $11,000 in 1990, $11 in 2000, and three cents in 2014. Google offers free storage for up to 15 GB of data.
Artificial intelligence relies fundamentally on Big Data, cheap storage, and cloud computing as its primary growth drivers. Over the last 20 years, storage and analytical capabilities have increased significantly. Artificial intelligence mirrors human cognitive functions like reasoning, learning, and problem-solving.
Unlike traditional technologies designed for specific execution tasks, artificial intelligence automates decision-making and handles unstructured challenges. Previous technologies assisted human direction by finding optimal paths for predefined goals. AI handles ambiguous situations, learns independently, and makes subjective decisions.
The United States Apollo 11 landing on the Moon on July 20, 1969, relied on rudimentary IT that was less powerful than a modern toaster. The Apollo Guidance Computer operated at 0.043MHz with approximately 64 KB of memory. The program involved 3,500 IBM employees using mainframes.
2. Reasoning Methods and Analytics Process
Data analytics relies on exploratory inductive reasoning and an iterative sequential cycle that requires close collaboration between domain experts and end users.
Deductive reasoning moves from general theories to specific observations to test hypotheses with certainty. Conversely, inductive reasoning observes specific data to discern patterns and form generalized theories. Because of this, Big Data insights can sometimes be impractical or flawed.
Unlike deductive reasoning which tests hypotheses, inductive reasoning is open-ended and exploratory. It works through pattern recognition on large datasets. Because of this, Big Data does not deliver 100 percent foolproof solutions and requires careful variable selection by managers.
Inductive Reasoning and Limitations
Many data scientists accept models that yield the best statistical results on a dataset without understanding cause and effect. Machine learning relies on pattern recognition, meaning it can pick up meaningless patterns unless inspected by human experts.
Data science algorithms can generate spurious correlations, making close collaboration with domain experts essential to verify intuitive sense.
Deep-learning algorithms have successfully matched or outperformed expert doctors in detecting skin cancer, heart arrhythmia, pneumonia, and breast cancer from medical images. Computers achieved 90 percent accuracy in distinguishing between viral and bacterial pneumonia in chest X-rays.
The Data Analytics Cycle
Data analytics is an evolutionary, iterative process rather than a rigid linear cycle. It evolves continuously with modifications at each step of development.
The process moves through specific operational stages:
- Unstructured Data
- Data Capture
- Data Sorting
- Structured Data
- Data Analysis
- Actionable Insights
- Predictive Models
- Future Forecast
- Results Evaluation
- Process Modifications
Extracting value from this cycle helps you make informed decisions while watching out for unintended second-order effects. The end user must remain at the core of the analytics process to ensure systems are easy to use and overcome institutional inertia.
You can guide this implementation through concrete operational steps:
- Revolve the entire process around the end user and build easy-to-use systems
- Define clear objectives, costs, and timelines as a starting point
- Choose the most appropriate database software, tools, and technology infrastructure
- Separate useful and implementable data using explicitly stated objectives
- Create algorithms through continuous communication between Big Data professionals and frontline teams
- Continuously evaluate maintainability and enhancements with business and IT teams
- Reassess the technology infrastructure to support an evolving development process
3. Applications in Business and Society
Organizations leverage Big Data and predictive analytics to secure competitive advantages, optimize operations, and drive innovation across multiple sectors.
Corporate and Platform Strategies
Companies utilize advanced data analytics, predictive algorithms, and personal information tracking not merely to sell goods, but to function fundamentally as data businesses. You can observe this shift clearly in global platforms that monitor user networks, locations, and tracking cookies across external websites.
Facebook amasses enormous personal data from over two billion monthly active users worldwide to monetize analytics for advertisers. Algorithms accurately predict intimate traits such as sexual orientation, political views, and emotional stability from simple user likes.
Amazon tracks customer purchases, online shopping carts, search queries, and even Kindle highlights to power personalized recommendations and an anticipatory shipping model. They also use individualized dynamic pricing based on user activity and competitor pricing to maximize profits.
McDonalds tracks point-of-sale data, video data, sensor data, and flow across drive-thru restaurants to make informed pricing, menu, and location decisions. In 2016, McDonald's recorded 24 billion dollars in annual revenues.
Data assets function as intangible assets that drive performance dispersion and create sustainable competitive advantages, even if their financial value is not immediately visible. Adam Saunders and Prasanna Tambe authored the paper Data Assets and Industry Competition: Evidence from 10-K Filings. They analyzed US firm filings from 1996 to 2012 and found that databases and data security investments compound in value over time.
An empirical study titled Big Data - Big Gains by Thomas Niebel, Fabienne Rasel, and Steffen Viete investigated the link between analytics and innovation intensity. The application of Big Data analytics is empirically linked to a higher propensity for firms to innovate and achieve greater market success through new products or services.
Healthcare and Public Administration
Big Data analytics accelerate genomics research by making the massive sequencing of human genes more affordable and efficient. Each human genome contains 20000 to 25000 genes, 3 million base pairs, and amounts to 100 gigabytes of data.
The cost of sequencing a human genome dropped from 40 million dollars when the Human Genome Project completed in April 2003 down to a consistent 5000 dollars. Two years prior to April 2003, sequencing cost 95 million dollars.
The SANSCOG project in Srinivasapura, Karnataka, aims to capture medical data from 10000 people, channeling terabytes of genetic and MRI data to a central supercomputing server. India has 2.4 percent of the world's land mass, 17 percent of the world's population, and a total population of 1.3 billion as referenced in UN data projections. One in 6 Indians is projected to be older than 60 by 2050, compared to 1 in 20 approximately 25 years prior.
Central banks utilize high-frequency granular data and automated algorithms to monitor financial markets and calibrate monetary policy during crises. The European Central Bank collects money-market statistical-reporting data from 52 largest euro-area banks, representing 80 to 85 percent of total balance sheet coverage. They track 10000 daily unsecured loan transactions worth 100 billion euros and 30000 secured loan transactions worth 500 billion euros.
Scientific evidence gathered through data analytics can help governments and NGOs design effective interventions for poverty alleviation and crime reduction. The Abdul Latif Jameel Poverty Action Lab network includes 158 affiliated professors across 51 universities.
J-PAL demonstrated that cognitive behavioral therapy reduced arrests per student by 12 percent and violent crime arrests by 20 percent in Chicago Public Schools, increased on-time high school graduation by 9 percent for US students receiving CBT, and deterred an average of 26 crimes per participant in Monrovia, Liberia.
Big Data analytics can provide a competitive edge in sports by optimizing team performance and individual player mechanics. The German team partnered with SAP SE and cut their average possession time from 3.4 seconds to 1.1 seconds during the 2014 FIFA World Cup in Brazil, featuring 32 final teams competing.
Personal location data and real-time Big Data analytics can help urban planners optimize crowd management, reduce congestion, and mitigate large-scale human movement disasters. On 29th September 2017, a stampede at the Elphinstone Road Railway Station foot over-bridge in Mumbai killed 22 people during peak hours and sudden rains. Earlier stampedes caused 37 fatalities at an Allahabad railway station in February 2013 and 2 fatalities at the New Delhi railway station in 2010.
Following the September 2010 earthquake, the Stronger Christchurch Infrastructure Rebuild Team developed a GIS viewer with nearly 600 interactive layers to guide the reconstruction of Christchurch, New Zealand.
4. Challenges, Failures, and Governance
Massive data collection introduces severe risks of privacy breaches, market monopolies, and prediction failures that demand strict cybersecurity and regulatory policies.
Analytics Failures and Organizational Hurdles
Data tools do not guarantee accurate predictions or substitute for conventional data collection. Google Flu Trends attempted to predict influenza activity by aggregating search queries. It chronically overestimated flu prevalence because people searching for symptoms were often ignorant of actual medical diagnoses or had different illnesses with similar symptoms. Google launched Google Flu Trends in 2008, but studies claimed it was wrong for 100 out of 108 weeks starting from August 2011.
Organizations adopting Big Data face constraints regarding human attention limits, talent shortages, software hype, and specific structural frictions in the retail industry. Herbert Simon noted that information abundance creates an attention scarcity. Organizations struggle with defining data collection goals, measuring return on investment, navigating vendor hype, and overcoming a shortage of deep analytical talent. Retailers specifically face hurdles in cross-channel customer recognition, dynamic pricing alignment, and personalized recommendations.
Blind reliance on Big Data analytics can be distorted by human cognitive biases, and domain experts often achieve superior results by integrating intuition. Users frequently suffer from confirmation bias, manipulating data parameters to fit preconceived notions or rejecting algorithm conclusions when they conflict with personal expertise. Human intuition remains critical for complex, high-speed judgments.
Privacy, Security, and Market Power
The opacity of Big Data analytics creates significant risks of unethical abuse, government surveillance, and corporate intrusion into personal privacy. Because data can be manipulated to support any agenda, aggressive profiling campaigns can compromise individual privacy rights without user consent or awareness. Target's predictive analytics successfully identified a teenage girl's pregnancy from her shopping items and mailed baby product coupons to her home before she informed her father during a 2012 incident involving a Target store in Minneapolis.
Once personal data is shared with corporations, you lose all control over its storage, use, and dissemination, creating severe risks of identity theft during cyberattacks. Companies often conceal breaches and pay hackers directly to maintain silence. Uber paid hackers $100,000 to conceal an October 2016 cyberattack that exposed data on 57 million riders and drivers, including names, email addresses, mobile numbers, and around 600,000 driver's license numbers.
Big Data industries operate on a winner-takes-all model where a slightly larger company rapidly eclipses and elbows out smaller competitors, preventing decentralized market coexistence. Unlike old-economy industries like manufacturing or pharmaceuticals which remain decentralized with many regional players, digital platforms benefit from network effects that concentrate market power in a handful of elite corporations. Google, Facebook, Amazon, and Microsoft dominate their sectors globally with no equivalent competitors holding comparable scale, unlike traditional auto companies such as Toyota, General Motors, and Ford.
India's Aadhaar project is the world's largest biometric identification system, but it faces intense legal challenges regarding privacy, state surveillance, and technological exclusion from welfare benefits. Established by the Unique Identification Authority of India, Aadhaar collects demographic and biometric data. The Supreme Court of India declared the right to privacy a fundamental right on August 24, 2017, affecting 1.34 billion people, while reports indicated around 135 million citizen records may have been leaked.
Data functions as a valuable intangible asset that is continuously built by tech companies through lowered short-term earnings, and it should be accounted for and taxed accordingly. Tech giants intentionally depress current profits by charging low prices or offering free services to accumulate proprietary data banks. Google provides search and storage for free, Amazon burns capital in markets like India, and Uber runs at a loss to capture user data trails, similar to how Facebook paid $22 billion to acquire WhatsApp in 2014.
Social media platforms pose a severe threat to democracy by exacerbating polarization and spreading misinformation. Platforms monetize public attention and use filtering mechanisms that create echo chambers, connecting users only with those who agree with them and distorting public opinion. Facebook acknowledged that between January 2015 and August 2017, 146 million users may have seen Russian misinformation, while YouTube admitted to 1,108 Russian-linked videos and Twitter to 36,746 accounts.
Online retailers use personal data to implement individual pricing rather than maintaining a general market equilibrium. Retailers have the power to create demand-supply equations for individual customers, adjusting prices dynamically based on factors like user history and perceived ability to pay. An airline or online retailer can quote higher prices to you based on your urgent need or immediate necessity, such as an emergency trip or a required college textbook.
Amazon's increasing reliance on robotics and massive e-commerce market share are driving retail job losses and shifting the economic landscape. Amazon's growth relies on heavy investments in automation, which replaces human retail employees while the company captures nearly half of the online retail market. An article in Quartz by Dave and Helen Edwards highlighted that there were 170,000 fewer retail jobs in 2017 alongside 75,000 more Amazon robots, while Amazon cleared $258.22 billion in US retail sales in 2018.
China's Social Credit System functions as a comprehensive mass surveillance and national reputation tool to rate and rank citizens and businesses. The system tracks and evaluates every behavior of a citizen, distilling it into a single number that determines eligibility for jobs, loans, and other services. The Chinese Government established a planning outline for the construction of the Social Credit System spanning from 2014 to 2020 to rate government affairs, commercial integrity, societal integrity, and judicial credibility.
Financial statements must account for non-cash expenses and timing mismatches to reflect true operational health and prevent accounting fraud:
- Allocate the cost of a tangible asset over its useful life using depreciation methods like straight-line or double-declining.
- Account for subscription income equally across the membership period rather than recognizing it all in the purchase year.
- Apportion and account for delayed payments or barter trade in financial statements across the appropriate periods.
10 Key Ideas
- Big Data encompasses massive datasets defined by the 3Vs of volume, velocity, and variety, driving exponential global growth and fueling artificial intelligence developments.
- Data analytics relies on exploratory inductive reasoning and an iterative sequential cycle that requires close collaboration between domain experts and end users.
- Data science algorithms can generate spurious correlations, making close collaboration with domain experts essential to verify intuitive sense.
- Organizations leverage Big Data and predictive analytics to secure competitive advantages, optimize operations, and drive innovation across multiple sectors.
- Companies utilize advanced data analytics, predictive algorithms, and personal information tracking not merely to sell goods, but to function fundamentally as data businesses.
- Big Data analytics accelerate genomics research by making the massive sequencing of human genes more affordable and efficient.
- Massive data collection introduces severe risks of privacy breaches, market monopolies, and prediction failures that demand strict cybersecurity and regulatory policies.
- Blind reliance on Big Data analytics can be distorted by human cognitive biases, and domain experts often achieve superior results by integrating intuition.
- Big Data industries operate on a winner-takes-all model where a slightly larger company rapidly eclipses and elbows out smaller competitors, preventing decentralized market coexistence.
- Data functions as a valuable intangible asset that is continuously built by tech companies through lowered short-term earnings, and it should be accounted for and taxed accordingly.