Book summary “Doing Data Science”: summary

Introduction

You live in an era where data accumulates faster than you can process it. The hype surrounding big data leaves you wondering what actually works. You face the challenge of extracting value from massive datasets without falling for algorithmic hubris or statistical traps. Rachel Schutt addresses this problem by capturing the birth of data science through conversations with pioneering practitioners at companies like Google, Facebook, and Yahoo.

Data science combines practical hacking and statistical inference to extract value from data while avoiding algorithmic hubris and data leakage.

The book guides you through this emerging field by moving from foundational statistical concepts to machine learning algorithms and practical engineering applications. You learn how to blend computer science and statistics to solve real problems.

• What is the difference between data science and traditional statistics?
• How do you build predictive models without overfitting?
• What role does engineering play in statistical inference?

1. Introduction to Data Science

Skepticism toward Big Data and data science is justified due to ambiguous definitions and extreme media hype. Buzzwords and unclear definitions create a high noise-to-signal ratio. Media narratives often ignore decades of foundational work by statisticians, computer scientists, and mathematicians.

Data science emerged from the convergence of hacking, statistics, and domain expertise to navigate the cultural shift of datafication.

Datafication and the Current Landscape

Datafication is the cultural and technological process of turning all aspects of life into quantifiable data. This concept was coined by Kenneth Neil Cukier and Viktor Mayer-Schoenberger in May 2013.

Google's augmented-reality glasses datafy the gaze, Twitter datafies stray thoughts, and LinkedIn datafies professional networks.

Data science combines practical hacking skills with theoretical statistical understanding. Mike Driscoll characterized data science as a blend of hacking and statistics by referencing Drew Conway's 2010 Venn diagram and Nathan Yau's 2009 post. The job title data scientist was coined in 2008 by DJ Patil at LinkedIn and Jeff Hammerbacher at Facebook.

Defining the Role

Data science is best executed by multidisciplinary teams whose collective profiles cover all necessary skills. You can explore these profiles through classroom exercises and meta-definitions.

Rachel Schutt handed out index cards in her Columbia class for 60 students to rate their skill levels on a relative scale and taped them to the blackboard. At the time of checking, there were 465 job openings in New York City alone for data scientists. Harlan Harris surveyed hundreds of data science practitioners in mid-2012 and used clustering to define subfields in Analyzing the Analyzers.

• Rate your skill levels on a relative scale across computer science, math, statistics, machine learning, domain expertise, communication and presentation skills, and data visualization.
• Riff on the profile by adding buckets or removing skills, using a different scale or visualization method, and considering the drawbacks of self-reporting.
• Start with a text-mining model by performing a Google search for data science to see what the masses define it as.
• Look at how practitioners describe what they do using word clouds.
• Apply a clustering algorithm to the input data of what people do to predict what field they belong to.

2. Statistical Inference and Exploratory Data Analysis

Statistical inference extracts meaning from data generated by random processes. Data represents traces of real-world processes shaped by subjective sampling methods.

Statistical inference and exploratory data analysis uncover patterns and uncertainties in data before building predictive models.

There are two sources of uncertainty. One is the randomness of the process itself. The other is the uncertainty of the data collection methods. Statistical estimators and mathematical models simplify these captured traces.

Proponents of Big Data argue that massive data eliminates the need for sampling because it records the truth. This assumption is flawed because data is never truly all-inclusive and often excludes marginalized groups. Internet surveillance misses sophisticated criminals. Polling misses those who did not vote.

Analyzing tweets immediately before and after Hurricane Sandy created the false impression that people were mostly shopping and partying. Coastal New Jersey residents dealing with severe property damage did not have time to tweet. Assuming data is objective can perpetuate historical biases in algorithms such as recruitment tools.

Statistical Inference and Models

A model is an artificial construction that abstracts reality. It removes extraneous details to represent uncertainty and patterns. Statistical models use mathematical functions containing parameters to capture data-generating processes. Building a model involves starting simply and writing down assumptions.

Exploratory Data Analysis

Exploratory Data Analysis is both a set of graphical tools and a mindset. It focuses on gaining intuition, debugging logs, and understanding data shape before building models. Developed by John Tukey at Bell Labs, it involves systematically plotting distributions without prior hypotheses. Daryl Pregibon and Diane Lambert performed EDA on Google-scale log data using 31 datasets representing simulated daily ad and click data from May 2012.

You can follow these steps to explore and summarize your data:

• Ask a question.
• Do background research.
• Construct a hypothesis.
• Test your hypothesis by doing an experiment.
• Analyze your data and draw a conclusion.
• Communicate your results.
• Create a new variable, age_group, that categorizes users into specific age buckets.
• Plot the distributions of number impressions and click-through-rate across categories.
• Define a new variable to segment or categorize users based on their click behavior.
• Explore the data and make visual and quantitative comparisons across user segments and demographics.
• Create metrics, measurements, and statistics that summarize the data such as CTR, quantiles, mean, median, variance, and max.
• Extend analysis across days to visualize metrics and distributions over time.

The data science process is a feedback loop where you move from raw data munging and EDA to modeling and deploying data products. Data products like recommendation systems create behavioral feedback loops by interacting with users. You must actively participate in every stage of this process, extending the traditional scientific method.

Doug Perlson founded RealDirect to address traditional brokerage inefficiencies and data lags. Traditional public real estate sales data has a 3-month lag. RealDirect applies data pooling and real-time interaction feeds to optimize real estate transactions and offer reduced commissions. Perlson ran the company with a subscription fee of $395 a month for sellers and a reduced commission rate of 2%.

You can apply these steps to housing data:

• Load in and clean up housing sales data.
• Conduct exploratory data analysis to find outliers and missing values, check date formats, and ensure numerical fields are treated correctly.
• Visualize and make comparisons across neighborhoods and across time.
• Summarize findings in a brief report aimed at the CEO.

3. Algorithms and Machine Learning

Algorithms form the basis of code efficiency, data preparation, and processing. You work with data engineering, optimization, and machine learning classes of algorithms.

Data science utilizes optimization, regression, and clustering algorithms while balancing statistical interpretability with predictive power.

Statisticians focus on real-world interpretation of parameters, capturing uncertainty with confidence intervals, and making explicit assumptions about data-generating distributions. Machine learning practitioners often treat predictive models as black boxes and utilize nonparametric solutions without explicit distribution assumptions.

Regression and Evaluation

Linear regression models the relationship between an outcome variable and one or more predictors by estimating parameters that minimize the sum of squared vertical distances. Least squares estimation calculates beta estimators by minimizing the residual sum of squares between observed outcomes and predicted values.

An analysis of user data on a social networking site uses age, income, and new friends to predict time spent on the site or revenue. The linear regression model predicting time spent from new friends yields an estimated intercept of -32.083, an estimated coefficient of 45.918, and a multiple R-squared value of 0.8244.

You evaluate linear models and classifiers using metrics such as R-squared, p-values, mean squared error, and cross-validation. Cross-validation by splitting data into an 80% training set and a 20% test set helps check if a model generalizes well and avoids overfitting.

Classification and Clustering

The k-nearest neighbors algorithm automatically classifies unassigned items by taking a majority vote of the most similar labeled items in feature space.

Predicting the credit rating category of a 57-year-old user with a 37,000 salary based on surrounding users leads to choosing an optimal $k$ value of 5 based on the lowest misclassification rate.

You implement k-NN through specific steps:

• Decide on your similarity or distance metric.
• Split the original labeled dataset into training and test data.
• Pick an evaluation metric.
• Run k-NN a few times, changing k and checking the evaluation measure.
• Optimize k by picking the one with the best evaluation measure.
• Create a new test set with unlabelled data to predict.

k-means is an unsupervised clustering algorithm that groups data points into $k$ bins by iteratively assigning points to centroids and updating centroid locations. The algorithm starts by randomly picking $k$ centroids in $d$-space, assigns each data point to the closest centroid, and then moves centroids to the average location.

Segmenting users based on ad impressions and number of clicks relies on this standard k-means approach, described by Hugo Steinhaus and Stuart Lloyd in 1957. James MacQueen first used the term k-means in 1967, and David Arthur and Sergei Vassilvitskii developed k-means++ in 2007.

4. Spam Filters and Naive Bayes

Linear regression is poorly suited for classifying binary outcomes like spam versus ham, and fails completely when the number of variables exceeds the number of observations. You cannot apply it directly to text data containing roughly 10,000 emails and 100,000 words.

Naive Bayes leverages probability and Laplace smoothing to build efficient text classifiers when standard regression and k-NN fail.

Trying to fit a linear model on a matrix of 10,000 email messages and 100,000 words results in a matrix that is far from invertible. The feature matrix becomes massively noninvertible and computationally intractable without severe feature selection.

Limitations of Standard Models

The k-nearest neighbors algorithm fails for high-dimensional text classification because sparse feature spaces make even nearest neighbors extremely far apart. While k-NN works well for low-dimensional problems like 256-dimensional handwritten digit recognition, applying it to a 100,000-dimensional word space triggers the curse of dimensionality.

Recognizing handwritten digits on a 16x16 grid yields a 256-dimensional space where k-NN achieves 97 percent accuracy, whereas it fails for high-dimensional text.

Naive Bayes and Data Wrangling

Bayes' Law provides a mathematical framework to compute the conditional probability of an event based on prior knowledge and observed evidence. Testing for a rare disease affecting 1 percent of the population requires a 99 percent sensitive and specific test to evaluate conditional probabilities correctly.

Naive Bayes builds an efficient text classifier by assuming word independence and aggregating individual word probabilities using counting and logarithms. Taking the logarithm converts products into sums, avoiding numerical underflow issues caused by multiplying tiny probabilities. Jake Hofman used bash command-line tools on Enron employee email data to count occurrences of words across 1,500 spam examples and 3,672 ham examples.

Laplace smoothing prevents zero probabilities in Naive Bayes by introducing pseudocounts as hyperparameters into word probability calculations. Adding terms like alpha and beta to numerator and denominator counts avoids overconfidence and extreme probability estimates of zero or one. You might set hyperparameters alpha equal to 1 and beta equal to 10, or choose small positive values like 1/5.

Data scientists must actively collect data using APIs, query languages, and web inspection tools when pre-packaged datasets are unavailable. Using the New York Times Developer API to fetch recent articles and Yahoo's YQL language lets you harvest information. When APIs are absent, tools like Firebug, curl, wget, and parsing libraries assist in extracting content.

Follow these steps when harvesting data:

• Check the terms and services of a website’s API before scraping.
• Register for an API key and review documentation.
• Inspect elements using browser extension tools to map HTML structures.
• Write shell scripts or python scripts using tools like curl, wget, grep, awk, or BeautifulSoup to harvest data.

5. Logistic Regression and Optimization

Logistic regression uses maximum likelihood estimation and convex optimization to solve binary classification problems in production environments.

Choosing a machine learning classifier is heavily dependent on operational constraints such as runtime, interpretability, and scalability rather than pure accuracy alone.

Credit card companies are legally required to explain denial-of-credit decisions, making interpretable models like decision trees preferable to black-box models. Media6Degrees updates bidding decisions using 500 models a day.

User browsing history can be transformed into a sparse matrix of hashed URLs to predict binary click outcomes, reducing ad targeting to a spam classification problem. Media6Degrees builds training sets where columns represent hashed URLs and the target variable is binary.

The Underlying Math

Logistic regression uses the inverse-logit function to transform linear predictors into values bounded between zero and one, which can be interpreted directly as probabilities. The typical order of magnitude for the click-through base rate is 1 percent. Estimating parameters requires maximum likelihood estimation and convex optimization algorithms because no closed-form solution exists.

Evaluation and Ranking

Models must be evaluated using metrics tailored to their specific context, such as AUC for ranking models, precision-recall for classification, and mean squared error for density estimation. A probability threshold of 0.5 is used to label an item as a click or non-click. Placing an ad for hair gel above an ad for deodorant may artificially increase hair gel clicks simply due to its higher position.

You must design validation experiments carefully when you build prototypes:

• Hold on to your existing best performer.
• Once you have a new idea to prototype, set up an experiment wherein the two best models compete.
• Rinse and repeat while not overfitting.

6. Time Stamps and Financial Modeling

Absolute timestamps, log returns, and exponential downweighting are essential for capturing trends and volatility in financial time series.

Absolute timestamps and chronological ordering are mandatory for capturing trends, seasonality, and true cause-and-effect relationships.

Ignoring timestamps prevents the detection of macro trends and seasonality. Predictive modeling requires strict causal modeling, meaning you must never use future information to predict the present. In an insulin injection example, tracking only relative time shows blood sugar dropping fifteen minutes post-injection, but fails to reveal an overall rising trend over several months.

Financial Data and Log Returns

Quantitative finance prefers log returns over percentage returns because log returns are additive and symmetric. A stock price dropping by 50 percent followed by a 200 percent increase returns to its origin; in log returns, this corresponds to -0.301 followed by +0.301. Financial and behavioral data must be properly transformed and normalized before modeling.

You must prepare the data using specific transformations:

• Normalize the data by subtracting the mean and dividing by the standard deviation.
• Alternatively normalize or scale by dividing by the maximum value.
• Take the log of the data.
• Bucket into evenly spaced or distributed buckets to create a categorical variable.
• Choose a meaningful threshold to transform the data into a new binary variable of 1 or 0.

Volatility and Regularization

Exponential downweighting compresses historical time series data into a running estimate by gradually discounting older information. Using a decay factor of 0.97 results in a half-life of roughly 23 periods. In low-information financial environments, linear regression combined with smoothness priors prevents overfitting caused by excessive degrees of freedom.

You can calculate this volatility estimate through concrete steps:

• Choose a decay factor representing the downweighting weight for older data.
• Calculate the new variance estimate by multiplying the old variance by the decay factor and adding the weighted new squared return.
• Define the minimization function incorporating the sum of squared errors.
• Add a penalty term for large coefficients to introduce a ridge-like regularization prior.
• Add a second penalty term utilizing a shift operator matrix to enforce the prior that consecutive coefficients vary smoothly.

7. Extracting Meaning from Data

Better features matter more than better algorithms. Will Cukierski emphasizes that raw logs require careful curation to build useful predictive inputs. Feature extraction turns raw event histories into structured datasets before training models.

Careful feature extraction, decision trees, and crowdsourced competitions significantly improve predictive model performance.

You record 30 days of user behavior for an app called Chasing Dragons by logging every action with timestamps. You brainstorm numerous features like visit days and points without worrying about redundancies. You process these event logs down to a structured dataset with rows as users and columns as features:

• Record each user behavior with timestamps during the data collection phase.
• Brainstorm a comprehensive list of features without worrying about redundancies or correlations.
• Process timestamped event logs down to a structured dataset with rows as users and columns as features.

Feature selection methods include filters, wrappers, and embedded techniques. You can use stepwise regression methods to systematically add or remove variables based on statistical criteria:

• Start with a regression model with no features and gradually add the best predictor one at a time in forward selection.
• Start with a regression model that includes all features and gradually remove the worst one at a time in backward elimination.

Decision Trees and Random Forests

Decision trees use entropy and information gain as an embedded feature selection method. Entropy measures how mixed up a dataset is. You build classification trees to predict user retention using structured steps:

• Compute entropy for the target variable to measure its degree of mixture.
• Calculate conditional entropy and information gain for each available attribute.
• Split the tree iteratively at the attribute that maximizes information gain until all points belong to the same class.

Random forests improve accuracy over single trees by combining bootstrap samples with random feature selection. You can configure this algorithm using specific steps:

• Take a bootstrap sample with replacement from the training dataset for each tree.
• Randomly select F features for each node within every tree.
• Apply the entropy-information-gain engine to decide which feature to split on at each stage.

8. Recommendation Engines

Matrix factorization and alternating least squares enable scalable recommendation engines by discovering latent features in user-item matrices.

Matt Gattis, co-founder and CTO of Hunch, used machine learning to generate personalized recommendations for various domains. The system found that asking users twenty specific questions allowed the engine to predict the remaining responses with high accuracy. Hunch relied heavily on cross-domain engineering and treating recommendation systems as a scalable data product.

Matrix Factorization and SVD

User-item interaction data can be modeled mathematically as a bipartite graph where users and items form separate sets of nodes connected by weighted edges representing preferences. Metadata about users and items can also be integrated into feature vectors to build a comprehensive user-item matrix.

The nearest neighbor algorithm suffers from several severe flaws at scale, including the curse of dimensionality, overfitting, feature correlation, and sparseness.

Singular Value Decomposition (SVD) decomposes an original rating matrix into three matrices, enabling dimensionality reduction through low-rank approximations. SVD factors a matrix X into U, S, and V-transpose, where S is a diagonal matrix containing singular values that measure the importance of each latent variable. By dropping less important singular values and their corresponding vectors, the dataset is compressed while retaining the core latent structure.

Principal Component Analysis and matrix factorization approximate the rating matrix using unobserved latent features that efficiently capture user taste. By minimizing the mean squared error between actual ratings and the dot product of user and item latent vectors, the algorithm discovers uncorrelated latent features. This bypasses the need for the diagonal matrix S found in standard SVD.

Alternating Least Squares

Alternating Least Squares (ALS) solves the non-convex matrix factorization optimization problem by iteratively optimizing user vectors while holding item vectors fixed, and vice versa. Because direct joint optimization lacks a simple closed-form solution, ALS alternates between fixing V to solve for U, and fixing U to solve for V. Because each user and item update depends only on known preferences, the updates can be easily parallelized across multiple machines.

You can configure the alternating least squares algorithm using specific steps:

• Pick a random V matrix.
• Optimize U while V is fixed.
• Optimize V while U is fixed.
• Repeat the previous two steps until coefficients change by less than epsilon.

10 Key Ideas

1. Data science emerged from the convergence of hacking, statistics, and domain expertise to navigate the cultural shift of datafication.
2. Statistical inference and exploratory data analysis uncover patterns and uncertainties in data before building predictive models.
3. Data science utilizes optimization, regression, and clustering algorithms while balancing statistical interpretability with predictive power.
4. Naive Bayes leverages probability and Laplace smoothing to build efficient text classifiers when standard regression and k-NN fail.
5. Logistic regression uses maximum likelihood estimation and convex optimization to solve binary classification problems in production environments.
6. Absolute timestamps and chronological ordering are mandatory for capturing trends, seasonality, and true cause-and-effect relationships.
7. Careful feature extraction, decision trees, and crowdsourced competitions significantly improve predictive model performance.
8. Matrix factorization and alternating least squares enable scalable recommendation engines by discovering latent features in user-item matrices.
9. Choosing a machine learning classifier is heavily dependent on operational constraints such as runtime, interpretability, and scalability rather than pure accuracy alone.
10. Assuming data is objective can perpetuate historical biases in algorithms such as recruitment tools.