Book summary “NoSQL distilled”: summary

Introduction

You face a difficult challenge when your modern web application grows to handle massive user loads. Traditional relational databases struggle to scale out across clusters of cheap servers, forcing you to maintain rigid schemas that slow down development.

NoSQL databases replace rigid relational schemas with specialized cluster-oriented data models to improve programmer productivity and data-access performance.

Pramod J. Sadalage and Martin Fowler explain how non-relational data stores handle scale and changing requirements. You will learn about aggregate data models, distribution mechanisms, consistency tradeoffs, and polyglot persistence.

You can answer several key questions as you read:

• How do you store data without rigid relational tables?
• When should you use document stores instead of graph databases?
• How do you handle consistency across distributed clusters?
• How do you migrate schemas without downtime?

1. Why NoSQL

Relational databases achieved long-term dominance by solving persistence, concurrency, and inter-application integration. They allow you to store large amounts of data in a backing store while using transactions to coordinate concurrent user access and error handling. They also serve as shared integration points for multiple applications and provide a mostly standard relational model.

The massive scale of 21st-century web properties forced a shift from scaling up to scaling out on clusters, exposing the limits of relational databases.

Relational databases are built on a shared-disk subsystem or require manual sharding, which introduces single points of failure, administrative hurdles, and high licensing costs. Large web pioneers like Google and Amazon published influential papers on BigTable and Dynamo, inspiring databases explicitly designed for clusters.

Relational Database Benefits

You encounter challenges when mapping rich domain models to relational tables, a friction known as impedance mismatch. Applications often use a shared database integration approach where multiple systems read and write the same tables. This practice creates tight coupling that slows down schema changes and deployment cycles.

Cluster Scaling Motivation

You can handle larger workloads by scaling up to bigger servers or scaling out across a cluster of commodity machines. Relational databases struggle with scaling out because their architecture relies on ACID transactions across single instances. Sharding splits your data across multiple nodes, but you must manage the routing and complex joins manually.

NoSQL Definition and Polyglot Persistence

NoSQL refers to an ill-defined set of open-source, nonrelational, cluster-oriented databases that give rise to polyglot persistence. The term originated from a 2009 meetup hashtag. Rather than meaning a strict opposition to SQL, NoSQL represents a movement where organizations use a mix of data storage technologies tailored to specific data needs, alongside a shift from integration databases to application databases wrapped in web services.

2. Aggregate Data Models

Aggregate-oriented databases group related information into complex records that serve as units for data manipulation, consistency, and cluster distribution.

Unlike relational databases that divide data into flat tuples, key-value, document, and column-family stores use aggregates. Derived from Domain-Driven Design, an aggregate acts as the natural unit for sharding and replication across cluster nodes, as well as the atomic boundary for updates. Relational databases remain aggregate-ignorant because they do not recognize these natural boundaries.

Aggregate Orientation Principle

You interact with an aggregate-oriented database by retrieving or storing entire structures rather than isolated rows. This design matches how you work with data in application code.

Key-Value and Document Models

A key-value store treats the aggregate as opaque, allowing quick lookups by key without internal querying. A document database exposes the internal structure of the aggregate, enabling partial retrievals, indexes, and queries based on fields inside the document.

Column-Family Store Mechanism

Influenced by Google Bigtable, column-family databases organize data into two-level maps of rows and column families to optimize clustered storage and access. They group columns into column families that are typically accessed together.

You can model these structures using distinct layouts:

• wide rows acting as lists with sort orders
• skinny rows acting as record types with fixed fields
• supercolumns providing nested maps

3. More Details on Data Models

Graph databases and schemaless storage provide flexible alternatives for highly connected relationships and variable data structures.

Graph Databases for Complex Relationships

Graph databases specialize in managing highly connected data by using nodes, edges, and relationship traversal rather than aggregates. While aggregate-oriented databases struggle with cross-aggregate references, graph databases make relationship traversal extremely cheap by shifting navigation overhead from query time to insert time. They typically run on single servers and support ACID transactions covering multiple nodes and edges.

Schemaless Storage and Implicit Schemas

Schemaless databases remove upfront database schemas and nonuniform data constraints, but rely on an implicit schema enforced by application code. Applications can freely add fields or handle variable structures without altering a fixed table structure. However, because code logic assumes specific field names and data types, an implicit schema exists within the application, shifting structural responsibility away from the database engine.

Materialized Views in NoSQL

Materialized views are precomputed and cached queries used in aggregate-oriented databases to efficiently answer questions that violate the primary aggregate structure. Because aggregate boundaries make certain analytical queries expensive, databases or applications maintain materialized views either eagerly during data updates or via batch jobs. This allows fast reads of derived data without scanning every aggregate.

4. Distribution Models and Consistency

Distributed databases balance replication and sharding while navigating the CAP theorem tradeoffs between consistency and availability.

Distribution Models

You can distribute data across multiple servers using two main paths. Replication copies the same data over multiple nodes. Sharding puts different data on different nodes.

Replication can be master-slave or peer-to-peer. Sharding balances the load and helps scale out writes and reads. Sharding alone can decrease resilience if a node failure makes that specific subset of data unavailable. In an ideal ten-server sharding setup, each server handles 10 percent of the load.

Update and Read Consistency

Distributed databases face write-write conflicts and read-write conflicts due to concurrency and replication delays. Pessimistic concurrency uses locks to prevent conflicts. Optimistic concurrency lets conflicts occur and detects them via conditional updates. Replication inconsistency happens when replicas receive updates at different times, leading to stale data until eventual consistency is reached.

Martin and Cindy see different hotel room availability due to replication delays between London and Boston while Pramod books the last room from Mumbai.

Systems mitigate these issues with specific consistency models:

• sequential consistency ensures all updates appear in the same order everywhere
• logical consistency maintains business invariants across operations
• read-your-writes consistency guarantees you see your own updates
• session consistency binds your session to a single replica
• sticky session routes your requests to the same node repeatedly

CAP Theorem and Quorums

The CAP theorem states that in a distributed system that may suffer network partitions, architects must trade off consistency against availability. Originally proposed by Eric Brewer and proved by Seth Gilbert and Nancy Lynch, CAP stands for Consistency, Availability, and Partition tolerance. In practice, developers often trade consistency for lower latency or higher availability depending on domain requirements.

Martin and Pramod try to book the last hotel room across a broken network connection between London and Mumbai.

Quorums allow systems to achieve strong consistency and resilience by involving a specific subset of nodes in read and write operations. A write quorum requires W > N / 2, meaning participating write nodes must form a majority of the replication factor N. Strongly consistent reads can be guaranteed if the read quorum R plus the write quorum W is greater than N. A 3 nodes replication factor is suggested as enough for good resilience.

Systems can also trade off durability for higher performance by keeping updates in memory and periodically flushing changes to disk. While losing data on a server crash is a risk, certain workloads like user-session states or telemetric data collection benefit from higher responsiveness by accepting non-durable writes.

5. Version Stamps

Long-running business transactions and peer-to-peer updates require version stamps and vector clocks to detect conflicts.

Business and System Transactions

Long-running business transactions cannot be locked inside short system transactions. You must rely on offline concurrency patterns like Optimistic Offline Lock. Applications use a version stamp to detect if underlying data has changed between the initial read and the final write. This check prevents stale updates.

HTTP etags used in web resources returning a 412 Precondition Failed HTTP response code demonstrate this mechanism when a resource has changed on the server. You can also implement compare-and-set or CAS operations.

Version Stamp Types

Version stamps can be implemented using several distinct schemes:

• counter
• GUID
• content hash
• timestamp
• composite stamp

Counters require a single master to avoid duplication but allow easy comparison of recentness. GUIDs and content hashes are globally unique and can be generated by anyone. They cannot be compared directly for recentness unless combined. CouchDB uses a combination of a counter and a content hash.

Vector Stamps

Peer-to-peer systems use vector stamps containing a set of counters for each node to track updates and detect write-write conflicts.

A vector stamp also known as a vector clock or version vector allows a system to determine if one version is newer. It reveals if two concurrent updates have created an unresolvable conflict because neither stamp's history contains the other. A 3 nodes example vector stamp format like [blue: 43, green: 54, black: 12] illustrates how distributed tracking functions.

6. Map-Reduce and Database Implementations

Map-reduce organizes cluster computations while specialized key-value, document, and column-family stores provide distinct operational characteristics.

Map-Reduce Pattern

The map-reduce pattern organizes cluster computations by combining map tasks that operate on single aggregates and reduce tasks that summarize values sharing a key. The map phase processes a single aggregate to emit key-value pairs, which allows high parallelism and locality of data access. The reduce phase combines multiple values sharing the same key. Partitioning allows reducers to run in parallel on different keys, while combiners reduce data volume before network transfer.

Analyzing customer orders to compute product revenue across sharded cluster nodes by mapping line items and reducing by product ID.

You should follow these practices for computations:

• Write a map function whose input is a single aggregate and output is key-value pairs.
• Write a reduce function that takes multiple map outputs with the same key and combines their values.
• Partition mapper output by key to run multiple reducers in parallel.
• Use a combiner function to cut down data transferred across the network by combining data for the same key.

Key-Value Stores

Key-value stores function as simple hash tables optimized for primary-key access, treating values as opaque blobs while offering high performance and easy scaling. API operations are limited to get, put, and delete by key. Because the database does not examine the value, applications must manage internal data structures. Consistency models are often eventually consistent, utilizing tunable quorum parameters like N, R, and W.

Storing user session data, shopping carts, and user preferences using unique keys inside buckets.

You can manage your data using these operations:

• Create buckets with default consistency values such as nVal, w, and r.
• Store objects via the store API or HTTP POST requests using unique keys.
• Fetch values using the fetch API or HTTP GET requests.

Document Databases

Document databases store self-describing, hierarchical tree data structures like JSON or BSON, allowing varying schemas within the same collection and examinable values. Unlike relational databases where every row requires a rigid schema, document databases permit flexible attributes and embedded child documents. They support rich query capabilities, horizontal scaling via sharding, and high availability through replica sets.

Storing customer profiles where one document contains addresses and another contains likes without schema violations.

You can organize your storage with these steps:

• Insert documents into a database and collection using commands like db.coll.insert(document).
• Configure replica sets and write safety levels using WriteConcern.
• Add read slaves and shard collections based on fields like firstname for write scaling.

Column-Family Stores

Column-family stores group keys mapped to values into multiple column families, organizing data as rows containing columns with names, values, and timestamps. The architecture features no master node, allowing any node to handle reads and writes. Data is initially written to a commit log and memtable, flushed to SSTables, and managed via configurable consistency levels and read repair.

Storing customer profiles in a Cassandra keyspace and column family with row keys and individual columns.

You can interact with the store through these methods:

• Create keyspaces and column families specifying comparators and validation classes.
• Insert and update column data using SET commands or Cassandra Query Language (CQL).
• Read data using GET commands or CQL SELECT statements.

7. Graph Databases, Migrations, and Polyglot Persistence

Graph databases store entities as nodes and their connections as directed edges with properties, allowing high-performance traversal across complex relationships. Unlike relational databases that calculate joins at query time, graph databases persist relationships as first-class citizens. This makes traversing connections extremely fast and enables flexible queries without changing the underlying node or edge models.

Graph databases excel at relationship traversals, while schema migrations and polyglot persistence coordinate multiple storage engines.

In Neo4j, nodes representing people like Martin and Pramod are connected by a FRIEND relationship edge which can hold properties such as since when they became friends. Graph databases support advanced traversal and pathfinding algorithms, as well as domain-specific query languages like Cypher and Gremlin. Queries can traverse graphs at arbitrary depths using directional filters and breadth-first or depth-first ordering. Pathfinding functions and algorithms like Dijkstra's can determine the shortest path or multiple distinct paths between two nodes. To find all paths between Barbara and Jill, graph algorithms utilize a PathFinder configured with an expander for types and maximum depth. Neo4j version 1.8 achieves high availability via replicated slaves.

Schema Migrations and Transition Phases

Relational databases manage schema changes through version-controlled sequential migration scripts, using transition phases and scaffolding to maintain backward compatibility. Deploying schema changes involves maintaining an ordered sequence of migration files tracked in a database changelog. For legacy environments with multiple dependent applications, transition phases use temporary views, triggers, and virtual columns to run old and new schemas in parallel. When renaming a column from fname to fullname, a BEFORE UPDATE trigger synchronizes data between both columns until all enterprise applications transition to the new schema.

You should follow these practices for schema changes:

• Store schema changes along with data migration scripts in version control.
• Name script files with incrementing sequential numbers reflecting database versions like 001_Description_Of_Change.sql.
• Maintain a ChangeLog table to track applied changes and automatically run missing scripts.

Schemaless NoSQL databases require careful schema management through application code parsing and incremental migration of existing data. Because application code expects a specific implicit schema, updating structures like renaming a field requires making read operations tolerant to old formats. Incremental migration updates data gradually over time when it is accessed and saves it back in the new format. When reading an order item, the application checks for the old price attribute and falls back to fullPrice if null, subsequently saving the record using the new attribute.

Polyglot Persistence Architecture

Polyglot persistence combines multiple specialized data storage technologies within a single application or enterprise to match varying data access requirements. Since different database engines solve different problems, forcing all data into a single RDBMS often leads to suboptimal performance. An e-commerce application uses a key-value store for shopping carts and session data before orders are confirmed and persisted into an RDBMS.

Wrapping individual data stores into dedicated services isolates dependent applications from database choices and centralizes data ownership. Instead of multiple applications communicating directly with diverse databases, wrapping each database in a service API allows the underlying data storage layers to evolve independently while maintaining unified access control and business logic. A graph database holding customer product purchase relationships is wrapped in a service so multiple enterprise applications can query it uniformly.

Theoretical reasoning is insufficient; you must actually build software using candidate databases to properly assess productivity and performance. To judge productivity, you should build representative features and rely on developer consensus regarding their experience in the environment. To judge performance, you must test representative scenarios, loads, and data volumes using resources like cloud computing to provision test clusters.

You can follow these testing practices:

• Run through current features and check how data usage fits potential data models.
• Take some initial features and build them using the technologies under consideration.
• Build the same features with a couple of different databases to see which works best.
• Test some of the bad fit cases to evaluate both the happy path and difficult paths.
• Build a representative subset of performance tests reflecting realistic loads and data volumes.

To hedge against uncertainty when choosing a database, you should encapsulate data access within explicit layers or services to make future migration easier. Using patterns like Data Mapper and Repository, or decomposing the database layer into services, reduces coupling. This allows you to replace a database or gradually introduce NoSQL into an existing application if initial choices do not work out.

You should apply these encapsulation rules:

• Encapsulate database choice by keeping all database code in a section of the codebase that is relatively easy to replace.
• Decompose the database layer into services that encapsulate data storage to reduce coupling.

10 Key Ideas

1. NoSQL databases replace rigid relational schemas with specialized cluster-oriented data models to improve programmer productivity and data-access performance.
2. Aggregate-oriented databases group related information into complex records that serve as units for data manipulation, consistency, and cluster distribution.
3. Graph databases and schemaless storage provide flexible alternatives for highly connected relationships and variable data structures.
4. Distributed databases balance replication and sharding while navigating the CAP theorem tradeoffs between consistency and availability.
5. Long-running business transactions and peer-to-peer updates require version stamps and vector clocks to detect conflicts.
6. Map-reduce organizes cluster computations while specialized key-value, document, and column-family stores provide distinct operational characteristics.
7. Graph databases excel at relationship traversals, while schema migrations and polyglot persistence coordinate multiple storage engines.
8. Relational databases manage schema changes through version-controlled sequential migration scripts, using transition phases and scaffolding to maintain backward compatibility.
9. Peer-to-peer systems use vector stamps containing a set of counters for each node to track updates and detect write-write conflicts.
10. Polyglot persistence combines multiple specialized data storage technologies within a single application or enterprise to match varying data access requirements.