# GCP SRE & Observability — Part 2: Capacity Planning, Autoscaling & Mitigating Incident Impact

> **Series:** GCP SRE & Observability (2 of 6) — aligned to Professional Cloud DevOps Engineer (PCDE) exam sections 3-5
> **Part 1:** `01-slis-slos-error-budgets-and-service-lifecycle.md` — SLIs, SLOs, Error Budgets & Service Lifecycle
> **Part 2:** This file — Capacity Planning, Autoscaling & Mitigating Incident Impact
> **Part 3:** `03-instrumenting-telemetry.md` — Instrumenting Telemetry
> **Part 4:** `04-cloud-logging-metrics-dashboards-and-alerting.md` — Cloud Logging, Metrics, Dashboards & Alerting
> **Part 5:** `05-distributed-tracing-and-troubleshooting.md` — Distributed Tracing & Troubleshooting Workflows
> **Part 6:** `06-performance-monitoring-and-finops.md` — Performance Monitoring & FinOps on GCP
> **Questions:** `questions.md`

> Assumes you're comfortable with Part 1's SLO/error-budget mechanism, with MIG/GKE/Cloud Run autoscaling configuration at the level covered in [GCP Cloud Engineer Foundations](/tutorials/gcp-cloud-engineer-foundations), and with the underlying capacity math (Little's Law, the HPA formula, load testing) covered in [Capacity Planning & Performance](/tutorials/capacity-planning-performance) — this chapter applies both to GCP's specific incident-response levers rather than re-deriving either.

## Table of Contents

1. [What This Chapter Covers](#what-this-chapter-covers)
2. [The Incident That Opens This Chapter](#the-incident-that-opens-this-chapter)
3. [Quotas, Limits, and Reservations — GCP's Capacity Planning Baseline](#quotas-limits-and-reservations--gcps-capacity-planning-baseline)
4. [Dynamic Workload Scheduler: Reserving Scarce Capacity Ahead of Time](#dynamic-workload-scheduler-reserving-scarce-capacity-ahead-of-time)
5. [Forecasting From Real Data With PromQL, Not Guesswork](#forecasting-from-real-data-with-promql-not-guesswork)
6. [Where Autoscaling Already Lives in This Site — A Deliberate Recap](#where-autoscaling-already-lives-in-this-site--a-deliberate-recap)
7. [Mitigating Impact, Lever 1: Draining and Redirecting Traffic](#mitigating-impact-lever-1-draining-and-redirecting-traffic)
8. [Mitigating Impact, Lever 2: Adding Capacity Fast](#mitigating-impact-lever-2-adding-capacity-fast)
9. [Mitigating Impact, Lever 3: Rollback Strategies](#mitigating-impact-lever-3-rollback-strategies)
10. [A Real Incident Playbook: All Three Levers Together](#a-real-incident-playbook-all-three-levers-together)
11. [Choosing the Right Lever — A Decision Framework](#choosing-the-right-lever--a-decision-framework)
12. [Terminology Map: Incident-Mitigation Levers Across AWS, Azure, and GCP](#terminology-map-incident-mitigation-levers-across-aws-azure-and-gcp)
13. [A Full Worked Example: Meridian's Incident Response Runbook](#a-full-worked-example-meridians-incident-response-runbook)
14. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
15. [Worked Practice Problems](#worked-practice-problems)
16. [Summary and What's Next](#summary-and-whats-next)

## What This Chapter Covers

PCDE's "managing service lifecycle" objective names capacity planning and autoscaling explicitly — but with GCP-specific vocabulary this site hasn't covered yet: quotas, reservations, and Dynamic Workload Scheduler. Its "mitigating incident impact" objective is narrower and more operational: three concrete levers — draining/redirecting traffic, adding capacity, and rollback — that a responder actually reaches for once an SLO's error budget (Part 1's mechanism) signals something is wrong. This chapter covers both objectives together, because in practice they're the same skill applied at two different timescales: capacity planning is deciding what headroom to have *before* an incident, and incident mitigation is deciding what to do with (or beyond) that headroom *during* one.

🎯 By the end of this chapter, you'll know which specific GCP mechanism to reach for when a service is failing right now — not the general theory of scaling (that's already covered) but the exact `gcloud` command, API call, or console action that drains traffic, adds capacity, or rolls back safely under real time pressure.

## The Incident That Opens This Chapter

Six weeks after Part 1's error-budget gate went live, it did exactly what it was built for — and Meridian's team learned it wasn't enough on its own. A regional weather event caused a sudden 4x spike in GPS-ingestion traffic as delivery drivers across the Southeast rerouted en masse, and Meridian's Pub/Sub-backed ingestion pipeline started backing up faster than its Cloud Run consumer could drain the subscription. `shipment-api`'s own error budget stayed healthy — the spike hit a different, upstream service — so Part 1's release gate had nothing to say about it at all. The on-call engineer, Devon, spent the first twelve minutes of the incident trying to remember whether the fix was "add more Cloud Run instances," "roll back last night's consumer deploy," or "just wait it out," because Meridian had never written down which lever applied to which symptom.

The incident resolved in nineteen minutes once Devon manually raised the Cloud Run consumer's max-instance ceiling — the right lever, reached by trial and error rather than a decision the team had already made in advance. **That's the gap this chapter closes**: Part 1 built the signal that says something is wrong; this chapter builds the decision tree for what to actually do about it once that signal fires, for the specific failure modes GCP's own incident-mitigation levers are built to address.

```mermaid
flowchart TD
    Spike(["4x traffic spike,<br/>GPS-ingestion Pub/Sub"]) --> Backlog{{"Subscription backlog<br/>growing faster than drain rate"}}
    Backlog --> Confused["12 min lost —<br/>no documented lever choice"]
    Confused --> Guess["Devon tries manual<br/>max-instances increase"]
    Guess --> Resolved(["Resolved — 19 min total"])

    classDef trigger fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef bad fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Spike,Backlog trigger
    class Confused bad
    class Guess,Resolved ok
```

**What to notice**: the fix that worked was correct, but arriving at it consumed most of the incident's total duration — the actual technical mitigation, once chosen, took under two minutes to apply. The lesson threaded through the rest of this chapter is that a documented, GCP-specific decision framework turns that twelve minutes of trial and error into a near-immediate, confident choice.

## Quotas, Limits, and Reservations — GCP's Capacity Planning Baseline

Capacity Planning & Performance already taught the *process* — forecast demand, plan for peak not average, decide on headroom. What that generic treatment couldn't cover is GCP's own hard ceiling underneath any capacity plan: **quotas**. A quota is a per-project (or per-region, per-resource-type) cap GCP enforces regardless of your billing account balance or your autoscaler's configuration — an autoscaling policy that wants to add a 51st Compute Engine instance when the project's quota caps at 50 doesn't get a 51st instance, it gets a failed API call, silently, unless someone is watching for it.

| Concept | What it actually limits | Where it's checked |
|---|---|---|
| **Quota** | A project or region's ceiling on a specific resource (CPUs, IP addresses, API requests/min) | Enforced synchronously on every resource-creating API call |
| **Limit** | A hard, non-negotiable system limit (not project-specific, can't be raised) — e.g., a MIG's maximum instance count | Enforced at the product/API level, not per-project |
| **Reservation** | Pre-purchased, guaranteed capacity for a specific machine type/zone, consumed automatically by matching VMs | Checked at instance creation; guarantees availability, not just permission |

```bash
# Check current CPU quota utilization in the region GPS-ingestion's
# Cloud Run consumer actually scales into — the check Devon should
# have run in minute one of the incident, not minute twelve
gcloud compute regions describe us-central1 \
  --project=meridian-shipment-prod \
  --format="table(quotas.metric, quotas.limit, quotas.usage)"
```

> [!IMPORTANT]
> A quota increase request is **not instant** — even an automatically-approved one typically takes minutes, and a request requiring manual Google review can take days. This is the single most important fact this chapter's incident-mitigation sections build on: **quota headroom has to already exist before an incident starts**, because requesting more mid-incident is a mitigation for the *next* incident, not this one.

**Cloud Quotas' Quota Adjuster** exists specifically to close that gap proactively: enabled at the project, folder, or organization level, it observes real usage trends and submits increase requests automatically *before* a project gets close to its ceiling, rather than waiting for a human to notice during a capacity review.

```bash
# Enable the quota adjuster for meridian-shipment-prod, so a traffic
# trend like this chapter's weather-event spike gets ahead of the
# quota ceiling before it becomes an incident, not during one
gcloud quotas adjuster settings update \
  --project=meridian-shipment-prod \
  --enablement=ENABLED
```

## Dynamic Workload Scheduler: Reserving Scarce Capacity Ahead of Time

For most Compute Engine capacity, on-demand provisioning plus a healthy quota ceiling is enough — GCP generally has the capacity, and the constraint is your own project's permission to consume it. For scarce, high-demand resources — GPUs and TPUs especially, and select VM families during regional capacity crunches — GCP itself may not have spare capacity available on demand even with unlimited quota. **Dynamic Workload Scheduler (DWS)** exists for exactly that scenario, in two modes suited to different capacity-planning shapes:

- **Flex-Start mode** — request a batch of scarce capacity for a bounded duration (1 minute to 7 days), fulfilled as soon as it becomes available, billed only for actual usage. Fits a training job or batch workload with a flexible start time but a real deadline.
- **Calendar mode** — reserve capacity for a specific future window, up to 90 days out, paying for the full reserved duration regardless of actual usage. Fits a planned, time-critical event where "as soon as available" isn't good enough — a product launch, a known seasonal peak.

```mermaid
quadrantChart
    title Choosing a GCP Capacity Mechanism
    x-axis Flexible timing --> Fixed, known timing
    y-axis Commodity capacity --> Scarce, high-demand capacity
    quadrant-1 Calendar mode
    quadrant-2 Standard reservations
    quadrant-3 On-demand headroom
    quadrant-4 Flex-Start mode
    route-optimizer GKE nodes:::commodity: [0.3, 0.15]
    A planned peak event:::scarce: [0.8, 0.75]
    A flexible ML training run:::scarce: [0.2, 0.8]
    shipment-api Cloud Run scaling:::commodity: [0.15, 0.1]

    classDef commodity color: #c3ccd4, radius: 8
    classDef scarce color: #6d43c0, radius: 10, stroke-color: #10161c, stroke-width: 2px
```

*Meridian's actual workloads sit mostly in the bottom-left — commodity Compute Engine and Cloud Run capacity with ordinary quota headroom is enough for everything this course's throughline runs. DWS becomes relevant the day Meridian's data team requests GPU capacity for a route-optimization model retraining job — the scarce, flexible-timing case Flex-Start mode fits exactly.*

## Forecasting From Real Data With PromQL, Not Guesswork

Capacity Planning & Performance's forecasting section already taught the *method* — pull real historical demand data, don't rely purely on trend extrapolation, plan against peak rather than average. The GCP-specific piece: **Cloud Monitoring now natively accepts PromQL queries against its own metric data**, including in the Metrics Explorer and alerting policies — the same query language [GCP Cloud Engineer Foundations Part 8](/tutorials/gcp-cloud-engineer-foundations/monitoring-logging-and-operations) introduced for Managed Service for Prometheus, usable against *any* Cloud Monitoring metric, not just Prometheus-sourced ones. MQL, Cloud Monitoring's older text-based query language, still works for existing dashboards but is no longer the recommended path for new work — PromQL is.

```promql
# Meridian's actual capacity-forecasting query ahead of a known seasonal
# peak: the 95th-percentile CPU utilization across gps-ingestion's MIG,
# over each of the past 12 weeks, to see the real trend rather than
# guess from the last few days alone
quantile_over_time(0.95,
  compute_googleapis_com:instance_cpu_utilization{
    resource.type="gce_instance",
    metadata.user_labels.app="gps-ingestion"
  }[12w:1d]
)
```

This is the query Priya's team now runs before every seasonal peak (the same weather-driven demand pattern that caused this chapter's opening incident recurs, less severely, every year around the same season) — a real 12-week p95 trend, not a felt sense of "traffic's been growing," is what actually justifies a pre-emptive MIG size floor or a Cloud Run `min-instances` bump ahead of the next predictable spike, closing the loop between this chapter's reactive incident-mitigation levers and genuinely proactive capacity planning.

> [!NOTE]
> Cloud Monitoring's own metric names follow the `compute_googleapis_com:instance_cpu_utilization`-style colon-separated PromQL naming convention for non-Prometheus-sourced metrics — a deliberate translation layer, not a coincidence, that lets the same query language work uniformly whether the underlying data came from a native Cloud Monitoring metric or an actual Prometheus scrape via Managed Service for Prometheus.

## Where Autoscaling Already Lives in This Site — A Deliberate Recap

This section is intentionally short, on purpose: MIG autoscaling policies (metric-based, predictive, scheduled) are covered in depth in [GCP Cloud Engineer Foundations Part 4](/tutorials/gcp-cloud-engineer-foundations/compute-engine-and-autoscaling), GKE's HPA/VPA/Cluster Autoscaler and Cloud Run's concurrency/instance model in [Part 5](/tutorials/gcp-cloud-engineer-foundations/gke-and-serverless), and the underlying scaling formula, cold-start problem, and load-testing methodology in [Capacity Planning & Performance Part 3](/tutorials/capacity-planning-performance/autoscaling-and-load-testing). Repeating any of that here would be exactly the duplication this course's scoping deliberately avoids.

| Autoscaling surface | Already covered in | What this chapter adds instead |
|---|---|---|
| MIG autoscaling policies | Foundations Part 4 | How to override it manually, fast, during an incident (next section) |
| GKE HPA/VPA/Cluster Autoscaler | Foundations Part 5, Capacity Planning Part 3 | How to drain a node/pod safely mid-incident without waiting for the autoscaler's own reaction time |
| Cloud Run concurrency/instances | Foundations Part 5 | How to raise `max-instances` as an emergency capacity lever, and the quota ceiling that still caps it |

> [!NOTE]
> If any of the terms in the left column feel unfamiliar, that's a real signal to read the linked chapter first — this chapter assumes you already know *how* each autoscaler works and focuses entirely on what to do when normal autoscaling reaction time isn't fast enough for the incident in front of you.

## Mitigating Impact, Lever 1: Draining and Redirecting Traffic

**Draining traffic away from an unhealthy backend is almost always the fastest available mitigation, because it doesn't require provisioning anything new — it just stops sending requests to the thing that's failing.** GCP exposes this mechanism differently depending on which compute platform is serving the traffic.

**Compute Engine behind an external/internal Application Load Balancer** — connection draining on the backend service delays removing an instance from rotation until its in-flight connections finish (or the timeout elapses), rather than cutting them off mid-request:

```bash
# Set a 60-second connection-draining window on shipment-api's backend
# service — long enough for an in-flight tracking request to finish,
# short enough not to stall an emergency instance removal
gcloud compute backend-services update shipment-api-backend \
  --global \
  --connection-draining-timeout=60
```

**Cloud Run** — traffic is already split by percentage across revisions; redirecting is an immediate `update-traffic` call, not a drain-and-wait:

```bash
# Immediately send 100% of traffic back to the last known-good
# revision — the fastest traffic-redirect GCP offers, since Cloud
# Run has no connection-draining wait built into this path at all
gcloud run services update-traffic shipment-api \
  --project=meridian-shipment-prod \
  --to-revisions=shipment-api-00042-abc=100
```

**GKE behind Gateway API** — an `HTTPRoute`'s `backendRefs` weights redirect traffic between Services with the same mechanism used for canary rollouts, just pointed in the opposite direction:

```yaml
# Redirect 100% of route-optimizer traffic away from a suspect new
# version back to the previous stable Service, by setting weights
# to the extremes rather than a canary's typical 95/5 split
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: route-optimizer-route
spec:
  rules:
    - backendRefs:
        - name: route-optimizer-stable
          port: 80
          weight: 100
        - name: route-optimizer-canary
          port: 80
          weight: 0
```

```mermaid
sequenceDiagram
    participant LB as Load balancer /<br/>Gateway / Cloud Run
    participant Old as Unhealthy backend
    participant New as Healthy backend

    Note over LB: Drain decision triggered<br/>(by alert or human)
    LB->>Old: Stop routing NEW connections
    LB->>Old: Existing in-flight requests<br/>allowed to finish (drain timeout)
    LB->>New: Route all new traffic here
    Old-->>LB: In-flight requests complete
    Old->>Old: Now safe to remove/recreate
```

**What to notice**: every platform's mechanism does the same two things in the same order — stop new traffic, let existing traffic finish — the only real difference is how fast that happens (Cloud Run's traffic-split update is near-instant; a Compute Engine connection drain waits out its configured timeout by design).

> [!TIP]
> **Best Practice**: set connection-draining timeouts *before* an incident, not during one — a 0-second default (draining disabled) is easy to forget was never configured until the moment an emergency instance removal cuts off real in-flight customer requests instead of waiting for them to finish.

## Mitigating Impact, Lever 2: Adding Capacity Fast

**Draining traffic helps when the fix is "stop sending requests to the broken thing." Adding capacity helps when the actual problem is "there isn't enough of the healthy thing."** GPS-ingestion's weather-event spike from this chapter's opening story is squarely this second case — nothing was unhealthy, there just wasn't enough Cloud Run consumer capacity to keep up with 4x normal volume.

```bash
# The fix Devon eventually found by trial and error — raised
# deliberately here as documented policy instead: bump Cloud Run's
# max-instances ceiling immediately, ahead of any autoscaler reaction
gcloud run services update gps-ingestion-consumer \
  --project=meridian-shipment-prod \
  --max-instances=200 \
  --concurrency=40

# The equivalent manual override for a Compute Engine MIG — bypasses
# the autoscaler's own target-tracking delay by setting size directly
gcloud compute instance-groups managed resize gps-ingestion-mig \
  --size=40 \
  --zone=us-central1-a
```

| Lever | Reaction time | Real constraint | Best fit |
|---|---|---|---|
| Raise Cloud Run `max-instances` | Seconds to take effect, minutes for cold-starts to catch up | Project quota ceiling on concurrent instances | A serverless service the autoscaler is already scaling, just not fast/high enough |
| Manual MIG resize | Seconds to trigger, minutes for new instances to boot and pass health checks | Regional quota, and Spot preemption risk if the overflow group is Spot | A MIG whose autoscaler's own target-tracking window is too slow for the spike's actual speed |
| Reservation/DWS Flex-Start burst | Minutes, dependent on actual capacity availability | Real physical GCP capacity — not guaranteed even with quota | Scarce resources (GPU/TPU) where on-demand capacity isn't guaranteed to exist |

> [!WARNING]
> Adding overflow capacity via Spot VMs during an incident carries the same preemption risk [GCP Cloud Engineer Foundations Part 4](/tutorials/gcp-cloud-engineer-foundations/compute-engine-and-autoscaling) already covered for routine load — except now it's happening *during* an active incident, where a mid-incident preemption compounds the outage instead of just costing a retried batch job. Prefer standard (non-Spot) overflow capacity for emergency capacity additions specifically, even at higher cost, unless the workload is genuinely fault-tolerant to a mid-incident restart.

🔍 **From the Trenches**: a different team at a company migrating a similar workload to GKE configured their overflow node pool entirely on Spot VMs to save cost, and during a real traffic-spike incident, the overflow nodes themselves got preempted nine minutes after joining the cluster — compute reclaimed by Google mid-incident, on top of the original capacity shortage. The immediate cause was choosing Spot for emergency headroom specifically because it's what the routine autoscaling config already used; the underlying condition was never distinguishing "Spot is fine for routine elastic scaling, because losing a Spot instance during normal operation is a minor, expected event" from "Spot is a real liability for *emergency* overflow capacity, because losing it while already degraded turns a manageable incident into a worse one." The fix was a separate, standard-VM overflow node pool reserved specifically for incident response, never touched during normal autoscaling.

## Mitigating Impact, Lever 3: Rollback Strategies

**When the actual root cause is "the last deploy introduced this," rollback is the correct lever — not draining, not adding capacity, since both of those treat the symptom while the bad code keeps running.** [GCP DevOps & CI/CD Platform Part 4](/tutorials/gcp-devops-cicd-platform/continuous-delivery-with-cloud-deploy) already built Cloud Deploy's automated rollback and canary-abort mechanics; this section is the incident-response-specific application of that same capability.

```bash
# Roll back the meridian-prod target to its last known-good release —
# the single command that undoes exactly what Part 4's Cloud Deploy
# pipeline just promoted, without touching draining or capacity at all
gcloud deploy targets rollback meridian-prod \
  --delivery-pipeline=shipment-api-pipeline \
  --region=us-central1
```

For Cloud Run services deployed outside a full Cloud Deploy pipeline (a hotfix path, say), the traffic-split mechanism from Lever 1 doubles as an instant rollback — redirecting 100% of traffic to the previous revision *is* the rollback, with no separate rollback API needed. For a GKE Deployment managed outside Cloud Deploy, `kubectl rollout undo deployment/route-optimizer` reverts to the prior ReplicaSet using Kubernetes' own revision history — the generic mechanism, not GCP-specific, but worth naming here because it's the one gap in an otherwise Cloud-Deploy-centric rollback story.

| Rollback surface | Command | What it actually reverts |
|---|---|---|
| Cloud Deploy target | `gcloud deploy targets rollback` | Redeploys the last successful release's exact artifact through the pipeline again |
| Cloud Run traffic | `gcloud run services update-traffic --to-revisions=OLD=100` | Instant traffic redirect — the "new" revision still exists, just receives 0% |
| GKE Deployment (unmanaged by Cloud Deploy) | `kubectl rollout undo deployment/NAME` | Reverts to the prior ReplicaSet from Kubernetes' own revision history |

> [!IMPORTANT]
> `gcloud deploy targets rollback` re-runs the **entire** delivery pipeline for the prior release — including any approval gates that release originally required — it does not silently bypass them just because it's an emergency. If `meridian-prod`'s approval gate is still configured, a rollback still needs that approval, which is the correct tradeoff (an unreviewed emergency rollback is itself a risk) but is worth knowing in advance so a responder isn't surprised mid-incident that the rollback "didn't happen instantly."

## A Real Incident Playbook: All Three Levers Together

Most real incidents don't cleanly fit one lever — Meridian's own weather-event spike started as a pure capacity problem, but the fix Devon eventually found (raising `max-instances`) took several minutes to fully take effect while the backlog kept growing, and the team briefly considered draining a fraction of GPS-ingestion traffic to a lower-fidelity degraded-mode consumer while capacity caught up.

```mermaid
stateDiagram-v2
    [*] --> Detected
    Detected --> Diagnose: SLO burn-rate alert fires (Part 1)
    Diagnose --> BadDeploy: Correlates with a recent release?
    Diagnose --> CapacityShort: Correlates with a traffic spike?
    Diagnose --> UnhealthyBackend: Correlates with a specific backend failing health checks?

    BadDeploy --> Rollback: Apply Lever 3
    CapacityShort --> AddCapacity: Apply Lever 2
    UnhealthyBackend --> Drain: Apply Lever 1

    Rollback --> Verify
    AddCapacity --> Verify
    Drain --> Verify
    Verify --> [*]: SLO burn rate returns to normal

    classDef detect fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef action fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef done fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Detected,Diagnose detect
    class BadDeploy,CapacityShort,UnhealthyBackend,Rollback,AddCapacity,Drain action
    class Verify done
```

**What to notice**: the diagnosis step is the one this chapter's opening incident actually skipped — Devon jumped straight to guessing a fix instead of first confirming *which* of the three trigger conditions actually applied. Part 5's troubleshooting workflow builds the systematic version of that diagnosis step; this chapter's contribution is making sure a lever exists and is documented for each of the three outcomes that diagnosis can produce.

## Choosing the Right Lever — A Decision Framework

| Symptom | Correlates with... | Reach for | Not this |
|---|---|---|---|
| Error rate spikes right after a deploy | A specific recent release | Rollback (Lever 3) | Adding capacity — the bad code just runs faster |
| Latency/errors rise with traffic volume, no recent deploy | A genuine demand spike | Add capacity (Lever 2) | Rollback — there's nothing to roll back to |
| One specific backend/instance/pod is failing health checks, others are fine | A localized backend failure, not systemic | Drain that backend (Lever 1) | A blanket capacity increase — it doesn't fix the one bad instance |
| Multiple signals present at once (deploy + traffic spike) | Ambiguous — diagnose before acting | Diagnose first via Part 5's workflow, then apply the matching lever(s) | Guessing and applying all three at once — makes root-causing the actual fix harder afterward |

> [!TIP]
> **Best Practice**: apply exactly one lever, verify its effect against the SLO's burn rate (Part 1), and only then consider a second lever if the first didn't fully resolve it. Applying rollback and a capacity increase simultaneously might resolve the symptom, but it leaves the postmortem unable to say which change actually fixed it — a real cost when the same failure mode recurs and the team needs to know which lever to reach for immediately next time.

## Terminology Map: Incident-Mitigation Levers Across AWS, Azure, and GCP

| Concept | GCP | AWS | Azure | Where the mapping breaks down |
|---|---|---|---|---|
| Connection draining | Backend service `connectionDrainingTimeout` | ALB/NLB target deregistration delay | Azure Load Balancer / Application Gateway connection draining | Near-identical concept and naming across all three — one of the cleanest mappings in this series |
| Scarce-capacity reservation | Dynamic Workload Scheduler (Flex-Start / Calendar mode) | EC2 Capacity Blocks for ML | Azure Reserved VM Instances (no Flex-Start-equivalent as-available mode) | GCP and AWS both now have a purpose-built as-available-or-scheduled mechanism for scarce (GPU/TPU) capacity; Azure's reservation model is closer to a pure long-term commitment without a Flex-Start-style bursty option |
| Pipeline-managed rollback | `gcloud deploy targets rollback` | CodeDeploy automatic rollback (on CloudWatch alarm) | Azure Pipelines release rollback | Conceptually aligned, though CodeDeploy's alarm-triggered auto-rollback is more automation-by-default than Cloud Deploy's typically human-invoked rollback command |
| Automatic quota headroom management | Cloud Quotas Quota Adjuster | Service Quotas (manual increase requests; no proactive adjuster equivalent as of this writing) | Azure subscription/resource quota increase (manual) | GCP is genuinely ahead here — Quota Adjuster's proactive, usage-trend-based auto-request has no direct AWS/Azure equivalent yet |

## A Full Worked Example: Meridian's Incident Response Runbook

The concrete artifact Meridian's team wrote after the weather-event incident — the decision tree from earlier in this chapter, turned into an actual on-call runbook entry:

```bash
# Meridian's GPS-ingestion incident runbook, condensed to command form
# — the exact sequence the team now follows instead of guessing

# 1. Confirm which trigger condition applies (diagnosis, per this
#    chapter's playbook state diagram)
gcloud logging read \
  'resource.type="cloud_run_revision" resource.labels.service_name="gps-ingestion-consumer" severity>=WARNING' \
  --project=meridian-shipment-prod --limit=50 --freshness=15m

# 2a. If it's a capacity shortfall (this incident's actual cause):
gcloud run services update gps-ingestion-consumer \
  --project=meridian-shipment-prod --max-instances=200

# 2b. If it's a bad deploy instead:
gcloud deploy targets rollback meridian-prod \
  --delivery-pipeline=gps-ingestion-pipeline --region=us-central1

# 2c. If it's one unhealthy backend among many:
gcloud compute backend-services update gps-ingestion-backend \
  --global --connection-draining-timeout=60

# 3. Verify: burn rate back under threshold within 10 minutes
gcloud monitoring slo describe gps-ingestion-availability \
  --service=gps-ingestion --project=meridian-shipment-prod
```

🧪 **Hands-on checkpoint**: run step 1's log query against a real (or practice) Cloud Run service under load and confirm you can distinguish a capacity-shortfall signature (rising request latency, no new deploy in the recent Cloud Build history) from a bad-deploy signature (errors starting at a timestamp matching a specific `gcloud deploy` rollout) before reaching for step 2 — the diagnosis step this chapter's opening incident skipped is exactly what this checkpoint exercises.

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | What to say/do instead |
|---|---|---|
| Requesting a quota increase *during* an incident and expecting immediate relief | Quota increases take minutes to days, not seconds | Provision quota headroom (or enable the Quota Adjuster) before an incident, as part of capacity planning |
| Using Spot VMs for emergency overflow capacity | A mid-incident preemption compounds an active outage | Reserve standard (non-Spot) capacity specifically for incident-response overflow |
| Applying rollback and a capacity increase simultaneously without diagnosing first | Makes it impossible to know afterward which change actually fixed it | Diagnose the trigger condition, apply one matching lever, verify, then escalate if needed |
| Assuming `gcloud deploy targets rollback` bypasses approval gates because it's an emergency | It re-runs the whole pipeline, including any configured approval gate | Know in advance whether the target has an approval gate, and treat rollback approval time as part of incident response time |
| Forgetting to configure connection draining before it's needed | A 0-second default cuts off in-flight requests during any backend removal, incident or not | Set a sensible drain timeout (30-60s typical) as a matter of routine configuration, not incident response |
| Treating DWS as a general capacity mechanism for everyday workloads | It exists for scarce, high-demand resources specifically (GPU/TPU, capacity-constrained regions) | Reach for standard on-demand provisioning plus quota headroom for ordinary Compute Engine/Cloud Run/GKE capacity |

## Worked Practice Problems

**Problem 1**: `shipment-api`'s error rate spikes at the same timestamp a new release finished deploying to `meridian-prod`, and separately, overall request volume is at a normal Tuesday-afternoon baseline. Which lever should Devon reach for, and which two levers should he specifically *not* reach for?

*Answer*: Rollback (Lever 3) — the timing correlation with a specific release, combined with normal (not elevated) traffic volume, rules out a capacity shortfall. He should not add capacity (there's no volume problem to add capacity for, and more instances just run the bad code faster) and should not drain a specific backend (the problem is the deployed code itself, not one unhealthy instance among healthy ones) — `gcloud deploy targets rollback` is the lever that actually addresses the root cause.

**Problem 2**: During an active incident, a responder wants to add overflow capacity to a GKE node pool and proposes using Spot VMs "since it's faster to provision and cheaper." What's the risk this chapter specifically warns about, and what should the responder do instead?

*Answer*: A Spot VM added as emergency overflow capacity can be preempted mid-incident, turning an already-degraded system into a worse one exactly when it can least afford a further capacity loss — the "From the Trenches" example in this chapter shows this happening nine minutes after the overflow nodes joined the cluster. The responder should provision standard (non-Spot) overflow capacity for incident response specifically, accepting the higher cost as the correct tradeoff during an active outage, reserving Spot for routine, fault-tolerant elastic scaling instead.

**Problem 3**: A team wants to use Dynamic Workload Scheduler for their everyday Cloud Run autoscaling, reasoning that "reserved capacity sounds more reliable than on-demand." Is this the right tool, and why or why not?

*Answer*: No — DWS exists specifically for scarce, high-demand resources (GPUs, TPUs, capacity-constrained regions) where on-demand provisioning isn't guaranteed to succeed even with sufficient quota. Cloud Run's everyday autoscaling draws from commodity, broadly available compute capacity, where standard on-demand provisioning plus adequate quota headroom (and the Quota Adjuster to keep that headroom current) is the correct, simpler mechanism — reaching for DWS here adds real complexity (Calendar mode's upfront payment for reserved duration, or Flex-Start's bounded-duration model) with no actual capacity-availability problem it's solving.

## Summary and What's Next

This chapter closed the gap Part 1's SLO/error-budget signal opened but didn't answer on its own: what to actually *do* once that signal fires. Quotas, limits, and reservations set the ceiling any capacity plan operates under; Dynamic Workload Scheduler reserves scarce capacity ahead of time for the cases on-demand provisioning can't guarantee; and the three incident-mitigation levers — draining traffic, adding capacity, and rollback — each map to a specific, diagnosable trigger condition rather than being interchangeable guesses. Meridian's weather-event incident, which once cost twelve minutes of confused trial and error, now has a documented runbook that gets a responder to the right lever in the time it takes to run one diagnostic query.

**Part 3** shifts from *reacting* to incidents toward *seeing* them coming sooner: instrumenting telemetry properly in the first place, with the Ops Agent, OpenTelemetry, Google Cloud Managed Service for Prometheus, and Cloud Service Mesh's automatic telemetry — the raw material every alert, every SLO, and every lever in this chapter ultimately depends on actually existing and being accurate.