Book summary “Site Reliability Engineering”: summary

Introduction

You face a constant struggle when growing a software service. Traditional system administration models require you to add staff linearly as traffic grows, which makes scaling expensive and unsustainable. Manual ticket handling and ad-hoc interventions drain engineering time and cause burnout.

Site Reliability Engineering solves this problem by treating operations as a software design challenge. You apply engineering principles to service management, using automation, error budgets, and data-driven monitoring to achieve high availability.

This book details how Google builds and operates reliable large-scale systems. You will learn how to balance feature velocity with operational stability. The following questions guide this exploration:

• How do you balance feature velocity with system reliability?
• What metrics define true user experience?
• How do you eliminate manual toil through software automation?
• How do you structure blameless postmortems and on-call rotations?

1. Introduction to Site Reliability Engineering

Traditional system administration scales linearly with service growth. This traditional approach creates direct financial costs and cultural tension between development and operations teams. Incentives conflict because developers want fast changes while operators want stability.

Site Reliability Engineering replaces manual operations with software engineering to scale systems efficiently while keeping operational load capped.

Ben Treynor Sloss founded Site Reliability Engineering at Google to treat system administration as human-computer engineering. Between forty and ninety percent of total costs of a system are incurred after birth. You must treat reliability as the most fundamental feature because a system is useless if users cannot access it.

SRE Versus Sysadmin

Site Reliability Engineering solves traditional operational limits by hiring software engineers to build automated systems for service management. At Google, fifty to sixty percent of Site Reliability Engineers are software engineers. Another forty to fifty percent are candidates with close software engineering qualifications and specialized UNIX and networking skills.

The 50 Percent Rule

You must cap aggregate operational work for Site Reliability Engineering teams at fifty percent.

An SRE team balances operational on-call duties and proactive software design.

An SRE team splits its time between these core activities:

• operate systems at a high level of abstraction using backup failsafes and thoughtful APIs;
• divide engineering time equally between operational on-call duties and proactive software design.

If operational work exceeds half of the team's time, excess tickets and operational overflow are redirected to the product development team. This mechanism acts as a safety valve, incentivizing developers to build operable systems that do not require constant manual intervention.

2. Embracing Risk and Error Budgets

Maximizing reliability indefinitely is counterproductive. SRE balances risk against innovation and cost using a non-linear risk continuum.

Error budgets establish an agreed-upon level of failure to balance reliability with feature release velocity.

Incremental improvements in reliability can cost orders of magnitude more due to redundant machine resources and opportunity costs. SRE aims to make a service reliable enough rather than one hundred percent reliable. You explicitly align the risk taken with the risk the business is willing to bear.

A service with one million dollars in revenue evaluating an availability target increase from 99.9% to 99.99% calculates the improved availability value as $900.

Risk Continuum

Since 100% reliability is practically impossible and yields no discernible user benefit due to external network noise, defining an availability target creates an error budget equal to 1 minus that target. You face a 0.01% budget for unavailability on a service with a 99.99% availability target.

Aligning Incentives

Product developers want fast release velocity while SREs want reliability. By establishing an error budget based on the SLO, both teams share a common incentive. When the budget is large, you can take risks and push quickly. When the budget is drained by outages or issues, releases are slowed down or halted.

3. Service Level Objectives and Indicators

Service level management uses quantitative indicators, objectives, and agreements to measure performance against user expectations.

SLI, SLO, and SLA

You must separate your terminology into distinct operational layers. A Service Level Indicator (SLI) is a quantitative measure of service performance like latency or error rate. A Service Level Objective (SLO) sets a target range for an SLI. A Service Level Agreement (SLA) adds explicit consequences, usually financial, if the objectives are missed.

Google Compute Engine maintains a 99.95% published target availability. A 99.99% availability target permits 52.56 minutes of allowable downtime in a year.

Setting effective SLO targets requires specific rules:

• Don't pick a target based on current performance
• Keep it simple
• Avoid absolutes
• Have as few SLOs as possible
• Perfection can wait

Percentiles Over Averages

Simple averages obscure tail latencies where a small percentage of requests take significantly longer due to queuing or variance. You should focus on high-order percentiles like the 99th or 99.9th percentile to capture plausible worst-case values.

A typical request service time of 50 ms can hide significant variance where 5% of requests run 20 times slower.

Google's global lock service Chubby used intentional controlled outages to flush out unreasonable dependencies when high reliability created a false sense of security.

4. Eliminating Toil

Toil is manual, repetitive operational work that scales linearly, and it must be strictly capped to preserve engineering time.

You encounter many tasks while running production services. Administrative chores like HR paperwork or team meetings are overhead rather than toil. Grungy work that yields permanent service improvements is not toil either.

Defining Toil

Work is toil if it meets specific operational criteria:

• human hands run an automated script;
• it is done repeatedly;
• a machine could do it;
• it is reactive rather than proactive;
• it expands linearly with traffic or users.

Engineering Activities

SRE teams must cap operational toil at a maximum of 50 percent of their time, dedicating the remaining half to engineering projects. By limiting toil to 50 percent and spending at least half their time on engineering, software, and systems development, you prevent burnout and avoid devolving into traditional operations teams.

Activities are divided into several distinct categories:

• software engineering involves writing code, automation scripts, and features;
• systems engineering involves configuration, tuning, capacity planning, and architecture consulting;
• toil consists of tactical, manual, and repetitive tasks with no enduring value;
• overhead includes administrative chores.

Because on-call shifts create a fixed floor of toil depending on rotation size, excessive toil must be actively managed. Quarterly surveys show that Google SREs report spending an average of 33 percent of their time toiling. Individual outlier SREs report toil extremes of 0 percent and 80 percent.

5. Monitoring Distributed Systems

Monitoring systems collect quantitative data to analyze long-term trends, compare experiment groups, alert on problems, build dashboards, and conduct ad hoc debugging. You can rely on white-box monitoring to inspect internal metrics like logs and endpoints. Black-box monitoring tests externally visible user behavior instead.

Effective monitoring tracks the four golden signals and categorizes outputs into urgent pages, tickets, and logging.

The four golden signals cover user-facing systems adequately:

• latency tracks the time it takes to service requests, separating successful and failed requests;
• traffic measures demand using a high-level metric like requests per second;
• errors track explicit, implicit, or policy-based failures;
• saturation measures how full system resources are and acts as an indicator of impending bottlenecks.

You must account for request distribution tails rather than focusing solely on averages. Averages hide extreme tail latencies where a small percentage of requests take significantly longer. To capture this without excessive storage costs, systems should collect request counts bucketed into exponential latency histograms.

Effective monitoring outputs must be strictly categorized into three types based on the urgency of human response required:

• pages: a human must do something now;
• tickets: a human must do something within a few days;
• logging: no one need look at this output immediately, but it is available for later analysis if needed.

Pages must be urgent, actionable, intelligent, and about novel problems, avoiding alert fatigue and prioritizing symptom-oriented monitoring. If a page requires only a robotic response or is triggered by a benign condition, it causes fatigue and ignores actual user impact.

6. The Evolution of Automation

Automation acts as a force multiplier for SREs, evolving through distinct stages from manual operations to fully autonomous systems.

Automation provides consistency, establishes a platform, reduces MTTR, speeds up action, and saves time. You can categorize the evolution of automation into five distinct hierarchical levels:

• No automation
• Externally maintained system-specific automation
• Externally maintained generic automation
• Internally maintained system-specific automation
• Systems that don't need any automation

Ads SRE automated database replica replacements and eventually built a failover daemon called Decider to run MySQL on Borg. This achieved a 95% success rate for Decider completing MySQL failovers in less than 30 seconds.

Prodtest and Idempotency

Prodtest extends unit testing frameworks to validate real-world service configurations, preventing deployment misconfigurations before they reach production.

To solve recurring misconfiguration issues across multiple clusters, SREs created Prodtest by extending the Python unit test framework to validate real-world services. Tests are paired with idempotent fix scripts that resolve failing conditions without damaging system configurations. Senior management gave a timeline of 3 months to reach network-ready on five new clusters, but you can achieve a 1 week target timeline for turning up five new clusters using automation.

• Run the cluster-specific test to validate service dependencies and configurations
• If a test like TestDnsMonitoringConfigExists fails, trigger the corresponding fix script like FixDnsMonitoringCreateConfig
• Scrape configuration from a database and check a skeleton configuration file into revision control
• Retry the test and proceed to push configurations if the test passes

A multi-petabyte Bigtable cluster configuration assumed disk layout behavior incorrectly and wiped all data, highlighting the danger of implicit safety signals in automation. Fix scripts could be safely re-run at a 15 minutes interval due to idempotency.

7. Release Engineering

Release engineering bridges software development and Site Reliability Engineering by defining repeatable, automated processes. Release engineering ensures software is built and delivered reliably through self-service models, high velocity, hermetic builds, and strict policy enforcement. You rely on four core principles: a self-service model for developer autonomy, high velocity via frequent small releases, hermetic builds that are completely isolated from the host machine environment, and strict access control policies.

Google release engineers developed tools like Rapid and Sisyphus to handle deployments safely and at scale.

Hermetic Builds

• Define build targets and dependencies using build configuration files
• Branch from mainline at a specific revision for releases without merging back
• Run automated unit tests and re-run them using the release branch to create an audit trail
• Assemble packages using Midas Package Manager (MPM) with unique hashes and signatures
• Deploy software using Rapid or Sisyphus rollout automation

Configuration Management

Configuration management must be handled carefully to prevent system instability, using strategies that decouple binaries from configurations when necessary. You can manage configuration files through several models depending on project needs: using the mainline directly, including configurations inside the MPM package with binaries, packaging configurations into separate MPM configuration packages for independent versioning, or reading configurations dynamically from external stores.

A feature implemented with a new binary can be released with a separate configuration package, allowing engineers to change flag settings like bad_quarto via cherry picking without rebuilding the binary.

• Evaluate project needs to choose between mainline configuration, bundled MPM packages, separate configuration MPM packages, or external stores
• Enforce strict code reviews for all configuration changes
• Use labels to fetch corresponding binary and configuration packages together

8. Simplicity and Stability

Boring code and minimal APIs eliminate accidental complexity and maintain system stability during progressive rollouts.

You must distinguish between essential complexity and accidental complexity. Every line of code added represents a liability and a potential vector for bugs. You should eliminate accidental complexity whenever possible.

Knight Capital experienced a catastrophic failure caused by dead code gated by a flag that was never disabled.

• Push back when accidental complexity is introduced into systems
• Constantly strive to eliminate complexity in onboarded systems
• Scrutinize code to ensure it drives business goals and routinely remove dead code
• Build bloat detection into all levels of testing

Virtue of Boring Code

Minimal APIs and loose coupling promote both developer agility and long-term system reliability. You can use protocol buffers to create wire formats that maintain backward and forward compatibility.

• Minimize the number of methods and arguments exposed in APIs
• Version APIs to allow consumers to upgrade safely and at their own pace
• Ensure distributed system components have clear and well-scoped purposes rather than combining unrelated functions

Gradient Descent Releases

Releasing many unrelated changes simultaneously obscures the cause of performance degradation or failures. You can use gradient descent to find optimal system performance by taking small, measurable steps.

Releasing 100 unrelated changes at once makes it exceptionally difficult to isolate which change impacted system performance.

• Break large updates into smaller, incremental release batches
• Measure and evaluate the performance impact of each individual code change in isolation

9. Being On-Call

Operational duties can easily consume an engineer's entire schedule if left unchecked. You must protect team capacity by strictly capping operational work at 50 percent of total time.

Sustainable on-call rotations require sufficient team size, incident limits, and regular preparation through disaster drills.

No more than 25 percent of an engineer's time can be spent on-call duties. The remaining operational time covers software releases and bug fixes.

Rotation Sizing

Maintaining a sustainable 24/7 on-call rotation requires sufficient team size to respect the 25 percent on-call time cap and prevent burnout. A single-site team with primary and secondary rotations needs a minimum of eight engineers. Each person is then on-call one week every month. Dual-site teams require a minimum of six engineers per site to maintain a follow the sun model.

Operational Overload

On-call shift quality is protected by limiting the number of incidents to ensure time for root-cause analysis. Google estimates that dealing with an incident takes an average of 6 hours. This establishes an upper bound of 2 incidents per day during a 12-hour shift. A sustainable rotation should maintain a median incident rate close to zero.

When operational load exceeds safe limits, you must take corrective action:

• Set concrete quarterly goals
• Fix noisy monitoring that causes alert fatigue
• Return the pager temporarily to developers

10. Effective Troubleshooting and Emergency Response

Troubleshooting distributed systems relies on the hypothetico-deductive method, combining system knowledge with telemetry to test potential failure hypotheses. You start with a problem report, inspect telemetry and logs, hypothesize potential causes, and test those theories by comparing observed states or actively treating the system.

When on-call for the Shakespeare search service, an engineer receives an alert for a black-box probe failure, uses curl to query the endpoint, and analyzes backend addresses from the response header to isolate the failing component.

During a major outage, the primary objective is to make the system work as well as possible under the circumstances rather than immediately finding the root cause.

Incident Command

Unmanaged incidents suffer from hyper-focus on technical tasks, poor communication, and freelancing. You can prevent chaos by applying principled incident management based on the Incident Command System.

• Declare an incident if you need to involve a second team, if the outage is visible to customers, or if the issue remains unsolved after an hour of concentrated analysis.
• Maintain a live incident document that is concurrently editable.
• Execute a clear handoff at the end of the working day by explicitly stating the transition and receiving firm acknowledgment.

A test blocking access to a single database out of a hundred within a MySQL database caused widespread outages of dependent services because rollback procedures were untested and flawed.

11. Postmortem Culture and Tracking Outages

Blameless postmortems and automated outage trackers turn failures into permanent architectural improvements.

Blameless Postmortems

Postmortems must be completely blameless to encourage engineers to surface failures without fear of punishment. A blameless postmortem assumes everyone involved had good intentions and acted correctly given their information. Instead of shaming individuals, the process investigates systemic and process flaws so engineers can design more resilient environments.

Healthcare and avionics industries use this approach where mistakes can be fatal, treating every error as an opportunity to strengthen systems.

To maintain this standard, you should follow specific steps:

• Focus on identifying contributing causes without indicting any individual or team.
• Assume good intentions and correct actions with available information.
• Investigate systemic reasons for incomplete or incorrect information.

Organizations must define objective criteria before incidents occur to decide when a postmortem is required. Setting predetermined triggers removes ambiguity and ensures significant events receive proper analysis:

• User-visible downtime or degradation beyond a certain threshold.
• Data loss of any kind.
• On-call engineer intervention such as release rollback or traffic rerouting.
• Resolution time above some threshold.
• Monitoring failure implying manual incident discovery.

Continuous cultivation keeps past lessons active. New SREs participate in the Wheel of Misfortune exercise where a previous postmortem is reenacted with engineers playing assigned roles.

Outage Tracking

An automated outage tracker passively receives all monitoring alerts to allow grouping, annotation, and analysis. Postmortems miss frequent, smaller-impact issues that do not trigger high-impact thresholds.

Grouping duplicate alerts into single incidents and applying semantic tags enables efficient operational analysis. Teams use tags like cause:network:switch, cause:network:cable, or customer-specific tags to categorize pain points.

• Combine multiple escalating notifications into a single incident entity.
• Apply free-form single-word tags with colons as semantic separators for hierarchical namespaces.
• Parse tags to link directly into bug tracking systems.

12. Testing for Reliability

Comprehensive testing combines traditional unit tests with continuous build systems, canary rollouts, and production probes.

Testing does not prove reliability, but failing tests prove the absence of it. Zero Mean Time to Repair bug detection blocks faulty pushes entirely. This increases the Mean Time Between Failures experienced by users.

Software Testing Types

Software testing is divided into traditional offline tests during development and production tests performed on live web services. Traditional tests form a hierarchy:

• unit tests;
• integration tests using dependency injection mocks;
• system tests like smoke tests, performance tests, and regression tests.

When you join projects with low test coverage, prioritize mission-critical code using these steps:

• Prioritize the codebase by stack-ranking components of the system by importance.
• Identify particular functions or classes that are absolutely mission-critical or business-critical.
• Identify which APIs are other teams integrating against.
• Create a series of smoke tests to run for every release.
• Document all reported bugs as test cases.

Continuous build systems running alongside versioned source control enforce that the latest code version is always working. Tools like Bazel build dependency graphs to run tests only on changed files. A service codebase might contain 21,000 simple tests.

Canary Tests

A canary test acts as structured user acceptance rather than deterministic testing. You expose a subset of servers to live traffic during progressive rollouts.

• Start by having the release impact 0.1% of user traffic.
• Scale by orders of magnitude every 24 hours while varying the geographic location of servers being upgraded (day 2: 1%, day 3: 10%, day 4: 100%).

By observing the cumulative number and rate of error reports during an exponential rollout, you can estimate the order of the underlying fault. Risky software must be isolated from user-facing production using deliberate architectural barrier defenses.

13. Handling Overload and Cascading Failures

Preventing cascading failures requires intent-based capacity planning, adaptive throttling, and strict retry budgets.

A cascading failure is an outage that grows over time because of positive feedback. When a single replica or server fails due to overload, it shifts extra load onto remaining infrastructure and creates a domino effect.

• Load test components until they break
• Test both gradual and impulse load patterns
• Test and understand how the component behaves as it returns to nominal load
• Test popular clients and their behavior regarding queues and backoff
• Test noncritical backends to ensure their unavailability does not interfere with critical components

The example Shakespeare search service production configuration can experience cascading failures when a cluster fails and forces all traffic onto another cluster.

Adaptive Throttling

When a backend rejects requests due to quotas or overload, processing the rejection can still consume significant CPU. Client-side adaptive throttling prevents backends from wasting resources on error rejections by having clients self-regulate outgoing traffic when rejections spike.

• Track the number of attempted application-layer requests and backend-accepted requests over a two-minute window at the client.
• Calculate the rejection probability once requests exceed the accepts multiplier threshold.
• Self-regulate by failing excess requests locally at the client without sending them to the network.

Retry Budgets

When small subsets of tasks are overloaded, immediate retries act as organic load balancing. If an entire cluster is struggling, blind retries amplify traffic and cause cascading failures. Restricting retries via per-request budgets, per-client ratios, and distinct error responses prevents combinatorial retry explosions during overloads.

• Enforce a per-request retry budget of up to three attempts before letting errors bubble up to the caller.
• Track the ratio of retry requests and only allow retries as long as this ratio is below 10%.
• Include a retry attempt counter in request metadata so backends can return an 'overloaded; don't retry' response when widespread congestion is detected.
• Retry failed requests only at the layer immediately above the failing component to avoid combinatorial retry explosions.

14. Data Integrity and Workflows

Users do not distinguish between data loss, corruption, and extended unavailability. To a user, data integrity without expected availability is effectively the same as having no data at all.

Protecting data at scale requires a multi-layered defense in depth strategy because no single mechanism can guard against all possible failure combinations.

Defense in Depth

Google SRE uses three successive layers of defense. You deploy soft deletion as the first line against user and developer errors, backups as the second line, and regular data validation as the third line.

During a 2011 Gmail incident, users experienced a 4-day outage where data was eventually proven safe, but the extended unavailability severely eroded user trust. Google Apps sets a starting threshold of 24 hours for downtime that is considered too long.

• Implement soft deletion or trash folders to allow recovery of inadvertently deleted items.
• Deploy tiered backups ranging from local snapshots to offsite tape storage.
• Establish regular out-of-band data validation checks to catch latent corruption.

Out-of-Band Validation

Automated validation pipelines run independently to verify invariants and prevent data degradation. Google Drive validators periodically check if file contents align with folder listings and automatically fix mismatches.

Refactoring the Gmail validator achieved a 60% reduction in disk spindle contention. The largest validator divides its workload using 10-14 shards.

• Implement validation job management.
• Set up monitoring, alerts, and dashboards.
• Provide rate-limiting features.
• Build troubleshooting tools and production playbooks.
• Create data validation APIs that make validators easy to add and refactor.

10 Key Ideas

1. Error budgets establish an agreed-upon level of failure to balance reliability with feature release velocity.
2. Work is toil if it is manual, repetitive, reactive, script-driven, and scales linearly with traffic or users.
3. Effective monitoring tracks the four golden signals and categorizes outputs into urgent pages, tickets, and logging.
4. Release engineering ensures software is built and delivered reliably through self-service models, high velocity, hermetic builds, and strict policy enforcement.
5. Sustainable on-call rotations require sufficient team size, incident limits, and regular preparation through disaster drills.
6. During a major outage, the primary objective is to make the system work as well as possible under the circumstances rather than immediately finding the root cause.
7. Blameless postmortems and automated outage trackers turn failures into permanent architectural improvements.
8. Comprehensive testing combines traditional unit tests with continuous build systems, canary rollouts, and production probes.
9. Preventing cascading failures requires intent-based capacity planning, adaptive throttling, and strict retry budgets.
10. Protecting data at scale requires a multi-layered defense in depth strategy because no single mechanism can guard against all possible failure combinations.