Book summary “Python machine learning”: summary

Introduction

You want to build intelligent systems that learn from data and make accurate predictions without manual rules. Writing these algorithms from scratch can feel overwhelming when you face complex mathematics, messy datasets, and rapidly changing frameworks.

Master machine learning and deep learning using Python, scikit-learn, and TensorFlow 2.

Sebastian Raschka provides a comprehensive guide that takes you from fundamental algorithms to advanced neural networks. You will learn how to preprocess data, evaluate models, and implement powerful techniques like convolutional networks and generative models.

The book answers several key questions to guide your journey:

• How do you implement core machine learning algorithms using Python?
• What are the best practices for cleaning and preparing raw datasets?
• How do you build, tune, and evaluate robust predictive models?
• When should you apply deep learning versus traditional supervised learning?

1. Foundations of Machine Learning

Machine learning enables systems to derive knowledge and make predictions from data without human-crafted rules. Artificial intelligence uses self-learning algorithms to process abundant structured and unstructured data. You can apply these predictive models to email spam filters and skin cancer detection.

Machine learning is broadly categorized into three types: supervised learning, unsupervised learning, and reinforcement learning.

Supervised learning uses labeled training data to predict outcomes. Unsupervised learning explores unlabeled data to discover hidden structures. Reinforcement learning trains an agent to maximize rewards through environmental interactions.

Types of Learning

Supervised learning comprises classification tasks for predicting discrete class labels and regression analysis for predicting continuous outcomes. Predicting math SAT scores based on study time serves as a standard regression example. Unsupervised learning allows the extraction of meaningful information and data compression. Marketers can discover customer groups based on interests without prior group membership knowledge.

Machine Learning Roadmap

Building a machine learning system follows a systematic roadmap covering data preprocessing, model training, cross-validation, and final evaluation. Raw data must be prepared before you train predictive algorithms.

Follow this roadmap to build a system:

• Preprocess raw data by extracting features, scaling them, and applying dimensionality reduction.
• Randomly divide the dataset into a separate training and test dataset.
• Compare a handful of different algorithms using a performance metric such as classification accuracy.
• Use cross-validation to estimate generalization performance and apply hyperparameter optimization to fine-tune the model.
• Evaluate the selected model on the test dataset to estimate the generalization error before making predictions on new data.

Python Tools

Python provides a robust ecosystem of specialized libraries for scientific computing and machine learning implementation. You can install packages using the terminal or package managers.

Execute these setup steps:

• Install Python version 3.7 or higher.
• Execute pip install SomePackage from the terminal to install additional packages.
• Update installed packages via the --upgrade flag using pip install SomePackage --upgrade.
• Use conda install SomePackage or conda update SomePackage when using the Anaconda distribution.

2. Training Simple Classifiers

Simple algorithms like the perceptron and Adaline learn optimal weights from data using gradient descent optimization.

Warren McCulloch and Walter Pitts published the MCP neuron model in 1943. It modeled biological brain cells as simple logic gates with binary outputs. Signals arrive at dendrites, integrate into the cell body, and generate an output along the axon if the signal exceeds a threshold.

Perceptron Learning Rule

Rosenblatt's perceptron rule automatically learns optimal weight coefficients to multiply with input features for classification. The perceptron computes a net input using a weight vector and features, passing it through a unit step function to output a binary class label of 1 or -1. You update weights simultaneously using a learning rate between 0.0 to 1.0 and the error between true and predicted labels.

Training a perceptron to classify Iris-setosa and Iris-versicolor flowers uses sepal length and petal length features.

Execute these practice steps to train the perceptron:

• Initialize the weights to 0 or small random numbers.
• For each training example, compute the output value.
• Update the weights simultaneously based on the perceptron learning rule.

Using NumPy for linear algebra operations speeds up computations through vectorization and hardware support. Unlike pure Python for loops, NumPy arithmetic operations are vectorized to utilize CPU SIMD support and rely on optimized C or Fortran libraries like BLAS and LAPACK. You can also apply the one-vs.-all technique to extend binary classifiers to handle multi-class classification problems.

Adaptive Linear Neurons

Adaline improves upon the perceptron by updating weights based on a continuous linear activation function and a differentiable cost function. Created by Bernard Widrow and Tedd Hoff, Adaline uses the sum of squared errors cost function instead of predicted class labels to compute model error, enabling the use of batch gradient descent.

Standardization shifts feature distributions to achieve zero-mean and unit variance, which helps gradient descent converge faster. By subtracting the sample mean and dividing by the standard deviation for each feature, the optimizer traverses the cost surface in fewer steps to find the global minimum.

Follow these steps for feature scaling standardization:

• Copy the feature matrix.
• Subtract the mean from each feature column and divide by its standard deviation.

Stochastic gradient descent updates weights incrementally for each training example, speeding up convergence on large datasets. Instead of accumulating errors across the entire dataset, SGD calculates the gradient per single example using mini-batch learning subsets of size 32. Shuffling training data every epoch prevents repetitive cycles and helps escape shallow local minima.

3. Classifiers Using scikit-learn

scikit-learn provides a robust tour of supervised classifiers ranging from logistic regression to support vector machines and random forests.

No single classification algorithm works best across all possible scenarios, making empirical comparison essential. According to David H. Wolpert's no free lunch theorem, algorithms make different assumptions and have unique quirks. You should compare performance across a handful of models considering feature counts, noise, and linear separability.

Training a supervised machine learning algorithm involves five main sequential steps:

• Selecting features and collecting labeled training examples.
• Choosing a performance metric.
• Choosing a classifier and optimization algorithm.
• Evaluating the performance of the model.
• Tuning the algorithm.

Stratification ensures that training and test subsets preserve the same class label proportions as the input dataset. Using the train_test_split function with stratification prevents biased splits by maintaining identical class distributions across subsets. You can split the Iris dataset into 70 percent training data (105 examples) and 30 percent test data (45 examples) using random_state=1.

Supervised Classifiers

Logistic regression uses the logistic sigmoid function to transform real-valued net inputs into probabilities between zero and one. The logit function is converted into an S-shaped sigmoid function that estimates the conditional probability of an example belonging to a class, which is then thresholded for binary classification.

Support Vector Machines maximize the distance between the decision boundary and the closest training examples. Maximizing the margin yields lower generalization error. For nonlinearly separable data, slack variables are introduced for soft-margin classification under cost penalization. You can train an SVM model with a linear kernel and C=1.0 to classify Iris flowers.

Decision trees recursively split nodes by maximizing information gain based on impurity measures like Gini impurity or entropy. At each step, you choose the feature split that results in the lowest child node impurity. Pruning max depth prevents overly deep trees from overfitting.

Regularization and Kernels

L2 regularization penalizes extreme parameter weights to prevent overfitting and handle feature collinearity.

By adding a regularization term scaled by a parameter inverse C, the model shrinks weight coefficients during training to improve generalization on unseen data. You can fit 10 logistic regression models with different values for the inverse-regularization parameter C.

The kernel trick projects linearly inseparable data into a higher-dimensional space where it becomes separable via a linear hyperplane. Instead of explicitly computing expensive high-dimensional feature mappings, kernel functions like the radial basis function calculate similarity scores between pairs of examples. You can train a kernel SVM with an RBF kernel, gamma=0.10, and C=10.0 on a synthetic XOR dataset of 200 examples.

Random forests combine multiple deep decision trees trained on bootstrap samples to reduce variance and prevent overfitting. Each tree is grown using a random bootstrap sample with replacement and a random subset of features at each node split, with predictions aggregated via majority vote. You can train a random forest from 25 decision trees using n_estimators=25 and n_jobs=2 with the Gini impurity measure.

K-nearest neighbors is a lazy, instance-based learning algorithm that memorizes the training dataset rather than estimating discriminative parameters. To classify new data points, the algorithm computes distances to training examples, finds the k nearest neighbors, and assigns a label via majority vote. You can implement a KNN model in scikit-learn using n_neighbors=5, p=2, and a Euclidean distance metric.

4. Building Good Training Datasets

Data preprocessing, handling missing values, scaling, and feature selection are critical steps for successful model training.

Missing values must be addressed before feeding data into computational tools because most algorithms cannot handle them or will produce unpredictable results. You can identify missing values using isnull() and sum() to count them per column.

• Identify missing values using isnull() and sum() to count them per column.
• Drop missing entries using dropna(axis=0) for rows or axis=1 for columns.
• Impute missing values using SimpleImputer with strategy set to mean, median, or most_frequent.

The Wine dataset consists of 178 wine examples with 13 features describing chemical properties, originating from the UCI machine learning repository. You can partition this dataset into separate training and test sets using train_test_split with test_size and stratify parameters to maintain class proportions.

Bringing features onto the same scale via normalization or standardization is crucial for optimization algorithms and distance-based estimators.

• Apply MinMaxScaler for min-max scaling to bound features in a [0, 1] interval.
• Apply StandardScaler to center feature columns at mean 0 with standard deviation 1.
• Fit scaling objects exclusively on the training data and transform the test data using those parameters.

Categorical features must be properly encoded into numerical values, distinguishing between ordinal features that carry order and nominal features that do not. T-shirt sizes like XL, L, and M represent ordinal features, while colors like red, green, and blue represent nominal features.

• Define a manual dictionary mapping for ordinal features and apply pandas map.
• Use LabelEncoder for encoding class labels into integers starting at 0.
• Apply OneHotEncoder or pandas get_dummies with drop_first=True to encode nominal features without creating multicollinearity.

Feature Selection

L1 regularization adds a penalty based on the sum of absolute weights to the cost function, promoting sparsity and serving as an automated feature selection method.

• Set penalty='l1' and solver='liblinear' in LogisticRegression to obtain sparse solutions.
• Adjust the inverse regularization parameter C to control regularization strength.

Sequential feature selection algorithms are greedy search techniques designed to reduce high-dimensional feature spaces into lower-dimensional subspaces while minimizing performance loss. Applying SBS with a KNN classifier on the Wine dataset identified a three-feature subset consisting of alcohol, malic acid, and OD280/OD315 of diluted wines.

• Initialize SBS with the full feature dimensionality.
• Determine the feature that maximizes the criterion function when removed.
• Remove the selected feature and repeat until the desired number of features is reached.

Random forests can assess feature importance by calculating averaged impurity decreases across all decision trees without assuming linear separability. Training a forest of 500 trees on the Wine dataset ranked proline, flavonoids, color intensity, OD280/OD315 of diluted wines, and alcohol as the top five most discriminative features.

• Fit a RandomForestClassifier with a specified number of estimators on training data.
• Access feature importances via the feature_importances_ attribute.
• Use SelectFromModel with a specified threshold to filter features based on importance.

5. Dimensionality Reduction

Feature selection algorithms preserve original features, but feature extraction transforms data into a new lower-dimensional space. You can use extraction to store increasing data volumes, improve computational efficiency, and reduce the curse of dimensionality for non-regularized models.

Dimensionality reduction compresses data subspaces using principal component analysis, linear discriminant analysis, and kernel methods.

PCA and LDA

Principal Component Analysis is an unsupervised linear transformation technique that identifies directions of maximum variance. You must standardize features beforehand because PCA is sensitive to data scales.

• Standardize the d-dimensional dataset.
• Construct the covariance matrix.
• Decompose the covariance matrix into its eigenvectors and eigenvalues.
• Sort the eigenvalues by decreasing order to rank the corresponding eigenvectors.
• Select k eigenvectors, which correspond to the k largest eigenvalues, where k is the dimensionality of the new feature subspace.
• Construct a projection matrix, W, from the top k eigenvectors.
• Transform the d-dimensional input dataset, X, using the projection matrix, W, to obtain the new k-dimensional feature subspace.

Linear Discriminant Analysis incorporates class label information by constructing scatter matrices.

• Standardize the d-dimensional dataset.
• For each class, compute the d-dimensional mean vector.
• Construct the between-class scatter matrix, S_B, and the within-class scatter matrix, S_W.
• Compute the eigenvectors and corresponding eigenvalues of the matrix S_W^(-1) * S_B.
• Sort the eigenvalues by decreasing order to rank the corresponding eigenvectors.
• Choose the k eigenvectors that correspond to the k largest eigenvalues to construct a d x k-dimensional transformation matrix, W.
• Project the examples onto the new feature subspace using the transformation matrix, W.

Kernel PCA

Kernel Principal Component Analysis uses the kernel trick to project nonlinearly separable data into higher dimensions. You apply nonlinear mappings combined with Radial Basis Function kernels to calculate similarities without computing coordinates explicitly.

• Compute the kernel matrix, K, using pairwise similarities for each pair of examples.
• Center the kernel matrix using the formula K' = K - 1_n*K - K*1_n + 1_n*K*1_n.
• Collect the top k eigenvectors of the centered kernel matrix based on their corresponding eigenvalues ranked by decreasing magnitude.

6. Model Evaluation and Workflows

Pipelines, cross-validation, and proper performance metrics ensure reliable model evaluation and hyperparameter tuning.

You chain data preprocessing transformers and a final estimator together to streamline workflows and prevent data leakage. The Pipeline class acts as a wrapper around intermediate transformer steps, such as StandardScaler and PCA, and a final estimator, such as LogisticRegression. Calling fit or predict passes the data sequentially through all intermediate steps before reaching the estimator.

Using make_pipeline with StandardScaler, PCA with 2 components, and LogisticRegression achieved a test accuracy of 0.956 on the Breast Cancer dataset after reducing 30 dimensions to 2 dimensions via PCA.

• Import make_pipeline, StandardScaler, PCA, and LogisticRegression.
• Create the pipeline object by passing transformers and the final estimator into make_pipeline.
• Call the fit method on the training data.
• Call the predict method or score method on the test dataset.

Pipelines and Cross-Validation

K-fold cross-validation provides a robust estimate of generalization performance by repeatedly splitting the training data into training and test folds. The training dataset is randomly split into k folds without replacement. In each iteration, k-1 folds are used for training, and the remaining single fold is used for evaluation.

Using 10-fold stratified cross-validation on the pipeline yielded a CV accuracy of 0.950 with a standard deviation of 0.014, where 10 folds serve as a good standard value for k.

• Import StratifiedKFold or cross_val_score from sklearn.model_selection.
• Specify the number of splits via n_splits or cv parameters.
• Iterate over the folds to train on training indices and evaluate on test indices.
• Calculate the mean and standard deviation of the resulting scores.

Evaluation Metrics

A confusion matrix reports counts of true positives, true negatives, false positives, and false negatives to detail classifier error types. By tabulating actual versus predicted class labels, a confusion_matrix function produces a square matrix showing where a classifier succeeds and fails.

Evaluating the SVM pipeline on the test set produced a confusion matrix with 71 true negatives, 40 true positives, 1 false positive, and 2 false negatives.

• Import confusion_matrix from sklearn.metrics.
• Generate predictions from a fitted estimator on test data.
• Call confusion_matrix with true labels and predicted labels.
• Visualize the matrix using Matplotlib matshow for clear interpretation.

7. Regression and Nonlinear Modeling

Regression models and decision trees capture continuous targets and handle nonlinear relationships in complex datasets.

Simple linear regression models the relationship between a single explanatory variable and a continuous response variable. You can explore the Housing dataset containing 506 examples and 13 features collected in 1978.

Ordinary least squares regression can be implemented using gradient descent by minimizing the sum of squared errors without a unit step function. You can diagnose regression models using residual plots and evaluate them quantitatively using mean squared error and the coefficient of determination. Regularization shrinks model weights and penalizes complexity to combat overfitting through Ridge regression, LASSO, and elastic net.

Linear Regression

The RANSAC algorithm fits a regression model to a subset of data designated as inliers to remain robust against outliers.

You can use RANdom SAmple Consensus to fit models reliably when your data contains severe contamination. The algorithm follows a strict sequence:

• Select a random number of examples to be inliers and fit the model.
• Test all other data points against the fitted model and add those points that fall within a user-given tolerance to the inliers.
• Refit the model using all inliers.
• Estimate the error of the fitted model versus the inliers.
• Terminate the algorithm if the performance meets a certain user-defined threshold or if a fixed number of iterations were reached; go back to step 1 otherwise.

Regularized and Nonlinear Regression

Polynomial regression introduces higher-degree polynomial terms to model nonlinear relationships while retaining a linear regression framework.

You can transform features to capture complex curves rather than straight lines. The transformation sequence involves specific steps:

• Add a second-degree polynomial term using PolynomialFeatures.
• Fit a simple linear regression model for comparison.
• Fit a multiple regression model on the transformed features for polynomial regression.
• Plot the results.

Random forest regression ensembles multiple decision trees using variance reduction to model complex nonlinear patterns without feature scaling.

8. Clustering Unlabeled Data

Unsupervised clustering algorithms group data into meaningful subgroups using k-means, hierarchical trees, and DBSCAN.

K-means is a prototype-based clustering algorithm that groups objects into spherical shapes by iteratively minimizing within-cluster variance.

K-means represents each cluster by its centroid, which is the average of continuous feature points. The optimization problem minimizes the within-cluster sum of squared errors, also known as cluster inertia.

A synthetic dataset of 150 randomly generated points grouped into three regions using make_blobs was clustered using KMeans.

You can implement the algorithm through specific steps:

• Randomly pick k centroids from the examples as initial cluster centers.
• Assign each example to the nearest centroid.
• Move the centroids to the center of the examples that were assigned to it.
• Repeat steps 2 and 3 until the cluster assignments do not change or a user-defined tolerance or maximum number of iterations is reached.

K-means++ improves initial centroid placement by choosing them far apart, resulting in more consistent and stable clustering outcomes. Instead of selecting random initial seeds which can lead to slow convergence or poor groupings, k-means++ uses a weighted probability distribution based on squared distances to pick initial centers.

Hierarchical and Density-Based Clustering

Agglomerative hierarchical clustering builds a bottom-up tree of clusters without requiring the number of clusters upfront. Starting with each example as an individual cluster, the algorithm iteratively merges the closest pairs based on linkage criteria until one single cluster remains.

DBSCAN identifies dense regions of points to discover arbitrary shapes and filter out noise without assuming spherical clusters. The algorithm categorizes points into core points, border points, and noise points based on neighborhood density within a specified radius and minimum sample threshold.

9. Ensemble Learning

Combining multiple classifiers via bagging, boosting, and majority voting drastically reduces overall error rates.

Combining independent classifiers through majority voting can drastically reduce the overall error rate compared to individual base classifiers. With 11 base classifiers each having an error rate of 0.25, the ensemble error drops to 0.034.

Majority Voting

A custom majority vote classifier can be implemented in Python using scikit-learn base classes and NumPy to combine class labels or predicted probabilities. You can build this meta-classifier by following these steps:

• Initialize the MajorityVoteClassifier with a list of classifiers, a voting strategy ('classlabel' or 'probability'), and optional weights.
• Implement the fit method using clone and LabelEncoder to ensure class labels start with 0.
• Implement the predict method to return majority voted class labels using NumPy bincount or argmax of probabilities.
• Implement the predict_proba method to return weighted average probabilities for each class.

Bagging and Boosting

Bagging reduces model variance by drawing bootstrap samples with replacement from the training dataset to train independent base classifiers. Applying BaggingClassifier with 500 decision trees to the Wine dataset classes 2 and 3 improved test accuracy from 0.833 to 0.917.

Adaptive Boosting trains weak learners sequentially, reweighting training examples in each iteration to focus on previously misclassified instances. Training an AdaBoostClassifier on 500 decision tree stumps using the Wine dataset achieved a training accuracy of 1.000 and a test accuracy of 0.917. You execute the boosting mechanics through specific operational steps:

• Set the weight vector w to uniform initial weights summing to 1.
• For each boosting round, train a weighted weak learner.
• Predict class labels and compute the weighted error rate.
• Compute the coefficient based on the error rate.
• Update the weights to increase the weight of misclassified examples and decrease the weight of correct ones.
• Normalize the weights to sum to 1 before the next round.

10. Neural Networks with TensorFlow

Multilayer perceptrons and TensorFlow 2 scale deep learning training efficiently across modern hardware accelerators.

Artificial neural networks originated in the 1940s, suffered interest loss due to training challenges for multi-layer architectures, and were revived by the 1986 popularization of backpropagation. Early foundational work by Warren McCulloch and Walter Pitts in 1943 was followed by Rosenblatt's perceptron in the 1950s. Interest waned during AI winters until D.E. Rumelhart, G.E. Hinton, and R.J. Williams repopularized backpropagation in 1986.

A modern GPU costs 65 percent of the price of an Intel Core i9-9960X CPU, has 272 times more cores, and is capable of around 10 times more floating-point calculations per second based on October 2019 data. You use tensors as symbolic handles for inputs and outputs in computation graphs.

Multilayer Perceptrons

A multilayer perceptron connects an input layer, one or more hidden layers, and an output layer, utilizing forward propagation and nonlinear activation functions to model complex functions. Backpropagation is a reverse-mode automatic differentiation technique that efficiently computes the partial derivatives of a complex cost function with respect to network weights.

TensorFlow APIs

You build models using the TensorFlow Keras API and construct efficient input pipelines:

• Create a dataset from existing tensors using tf.data.Dataset.from_tensor_slices.
• Combine two tensors into a joint dataset using tf.data.Dataset.zip or from_tensor_slices with a tuple.
• Apply transformations to dataset elements using the .map method.
• Shuffle elements by calling .shuffle with a specified buffer_size.
• Create batches by calling the .batch method.
• Repeat a dataset for multiple epochs using the .repeat method.

11. Image Classification with CNNs

Convolutional neural networks extract hierarchical visual features to achieve state-of-the-art image classification accuracy.

Inspired by the human visual cortex discovered by David H. Hubel and Torsten Wiesel, CNNs use sparse connectivity and parameter-sharing instead of full connections to reduce weights and capture salient local patterns effectively. David H. Hubel and Torsten Wiesel inserted a microelectrode into the primary visual cortex of an anesthetized cat in 1959 to observe how neurons respond to light patterns.

Convolution Mechanics

Discrete convolutions and cross-correlations use kernels, strides, and padding modes to slide across inputs and control output feature map dimensions. Most deep learning frameworks implement cross-correlation without rotating the filter matrix but refer to it as convolution. The three padding modes are full, same, and valid, with same padding being preferred to preserve spatial dimensions.

Deep CNNs and Data Augmentation

Data augmentation techniques artificially expand limited training datasets through geometric and color transformations to reduce overfitting. Operations such as random bounding box cropping, horizontal flipping, contrast adjustment, and brightness adjustment modify training images dynamically while evaluation data uses center crops. Building a gender classifier on 16,000 training face images from the CelebA dataset using data augmentation and achieving a 94.75% test accuracy.

You can implement this workflow in TensorFlow Keras:

• Load the CelebA dataset using tfds.builder with celeb_a.
• Take 16000 training examples and 1000 validation examples.
• Define a preprocess function that takes image and attribute dictionaries.
• Apply random crop, resize, and random flip left-right for training mode, and bounding box cropping for evaluation mode.
• Normalize pixel values by dividing by 255.0.
• Construct a CNN with four convolutional layers (32, 64, 128, and 256 filters) interspersed with max-pooling and dropout layers.
• Add a GlobalAveragePooling2D layer to reduce spatial dimensions.
• Add a final Dense layer with a single output unit and no activation.
• Compile the model with the Adam optimizer and BinaryCrossentropy with from_logits=True.
• Train the model using fit and evaluate on the test dataset.

10 Key Ideas

1. Machine learning uses self-learning algorithms to process abundant structured and unstructured data across supervised, unsupervised, and reinforcement learning types.
2. Simple algorithms like the perceptron and Adaline learn optimal weights from data using gradient descent optimization.
3. scikit-learn provides a robust tour of supervised classifiers ranging from logistic regression to support vector machines and random forests.
4. Data preprocessing, handling missing values, scaling, and feature selection are critical steps for successful model training.
5. Dimensionality reduction compresses data subspaces using principal component analysis, linear discriminant analysis, and kernel methods.
6. Pipelines, cross-validation, and proper performance metrics ensure reliable model evaluation and hyperparameter tuning.
7. Regression models and decision trees capture continuous targets and handle nonlinear relationships in complex datasets.
8. Unsupervised clustering algorithms group data into meaningful subgroups using k-means, hierarchical trees, and DBSCAN.
9. Combining multiple classifiers via bagging, boosting, and majority voting drastically reduces overall error rates.
10. Multilayer perceptrons and TensorFlow 2 scale deep learning training efficiently across modern hardware accelerators.