Capacity Planning, Autoscaling & Mitigating Incident Impact

Assumes you're comfortable with Part 1's SLO/error-budget mechanism, with MIG/GKE/Cloud Run autoscaling configuration at the level covered in GCP Cloud Engineer Foundations, and with the underlying capacity math (Little's Law, the HPA formula, load testing) covered in Capacity Planning & Performance — this chapter applies both to GCP's specific incident-response levers rather than re-deriving either.

Table of Contents#

1. What This Chapter Covers
2. The Incident That Opens This Chapter
3. Quotas, Limits, and Reservations — GCP's Capacity Planning Baseline
4. Dynamic Workload Scheduler: Reserving Scarce Capacity Ahead of Time
5. Forecasting From Real Data With PromQL, Not Guesswork
6. Where Autoscaling Already Lives in This Site — A Deliberate Recap
7. Mitigating Impact, Lever 1: Draining and Redirecting Traffic
8. Mitigating Impact, Lever 2: Adding Capacity Fast
9. Mitigating Impact, Lever 3: Rollback Strategies
10. A Real Incident Playbook: All Three Levers Together
11. Choosing the Right Lever — A Decision Framework
12. Terminology Map: Incident-Mitigation Levers Across AWS, Azure, and GCP
13. A Full Worked Example: Meridian's Incident Response Runbook
14. Common Mistakes and Interview Traps
15. Worked Practice Problems
16. Summary and What's Next

What This Chapter Covers#

PCDE's "managing service lifecycle" objective names capacity planning and autoscaling explicitly — but with GCP-specific vocabulary this site hasn't covered yet: quotas, reservations, and Dynamic Workload Scheduler. Its "mitigating incident impact" objective is narrower and more operational: three concrete levers — draining/redirecting traffic, adding capacity, and rollback — that a responder actually reaches for once an SLO's error budget (Part 1's mechanism) signals something is wrong. This chapter covers both objectives together, because in practice they're the same skill applied at two different timescales: capacity planning is deciding what headroom to have before an incident, and incident mitigation is deciding what to do with (or beyond) that headroom during one.

🎯 By the end of this chapter, you'll know which specific GCP mechanism to reach for when a service is failing right now — not the general theory of scaling (that's already covered) but the exact gcloud command, API call, or console action that drains traffic, adds capacity, or rolls back safely under real time pressure.

The Incident That Opens This Chapter#

Six weeks after Part 1's error-budget gate went live, it did exactly what it was built for — and Meridian's team learned it wasn't enough on its own. A regional weather event caused a sudden 4x spike in GPS-ingestion traffic as delivery drivers across the Southeast rerouted en masse, and Meridian's Pub/Sub-backed ingestion pipeline started backing up faster than its Cloud Run consumer could drain the subscription. shipment-api's own error budget stayed healthy — the spike hit a different, upstream service — so Part 1's release gate had nothing to say about it at all. The on-call engineer, Devon, spent the first twelve minutes of the incident trying to remember whether the fix was "add more Cloud Run instances," "roll back last night's consumer deploy," or "just wait it out," because Meridian had never written down which lever applied to which symptom.

The incident resolved in nineteen minutes once Devon manually raised the Cloud Run consumer's max-instance ceiling — the right lever, reached by trial and error rather than a decision the team had already made in advance. That's the gap this chapter closes: Part 1 built the signal that says something is wrong; this chapter builds the decision tree for what to actually do about it once that signal fires, for the specific failure modes GCP's own incident-mitigation levers are built to address.

What to notice: the fix that worked was correct, but arriving at it consumed most of the incident's total duration — the actual technical mitigation, once chosen, took under two minutes to apply. The lesson threaded through the rest of this chapter is that a documented, GCP-specific decision framework turns that twelve minutes of trial and error into a near-immediate, confident choice.

Quotas, Limits, and Reservations — GCP's Capacity Planning Baseline#

Capacity Planning & Performance already taught the process — forecast demand, plan for peak not average, decide on headroom. What that generic treatment couldn't cover is GCP's own hard ceiling underneath any capacity plan: quotas. A quota is a per-project (or per-region, per-resource-type) cap GCP enforces regardless of your billing account balance or your autoscaler's configuration — an autoscaling policy that wants to add a 51st Compute Engine instance when the project's quota caps at 50 doesn't get a 51st instance, it gets a failed API call, silently, unless someone is watching for it.

# Check current CPU quota utilization in the region GPS-ingestion's
# Cloud Run consumer actually scales into — the check Devon should
# have run in minute one of the incident, not minute twelve
gcloud compute regions describe us-central1 \
  --project=meridian-shipment-prod \
  --format="table(quotas.metric, quotas.limit, quotas.usage)"

Cloud Quotas' Quota Adjuster exists specifically to close that gap proactively: enabled at the project, folder, or organization level, it observes real usage trends and submits increase requests automatically before a project gets close to its ceiling, rather than waiting for a human to notice during a capacity review.

# Enable the quota adjuster for meridian-shipment-prod, so a traffic
# trend like this chapter's weather-event spike gets ahead of the
# quota ceiling before it becomes an incident, not during one
gcloud quotas adjuster settings update \
  --project=meridian-shipment-prod \
  --enablement=ENABLED

Dynamic Workload Scheduler: Reserving Scarce Capacity Ahead of Time#

For most Compute Engine capacity, on-demand provisioning plus a healthy quota ceiling is enough — GCP generally has the capacity, and the constraint is your own project's permission to consume it. For scarce, high-demand resources — GPUs and TPUs especially, and select VM families during regional capacity crunches — GCP itself may not have spare capacity available on demand even with unlimited quota. Dynamic Workload Scheduler (DWS) exists for exactly that scenario, in two modes suited to different capacity-planning shapes:

• Flex-Start mode — request a batch of scarce capacity for a bounded duration (1 minute to 7 days), fulfilled as soon as it becomes available, billed only for actual usage. Fits a training job or batch workload with a flexible start time but a real deadline.
• Calendar mode — reserve capacity for a specific future window, up to 90 days out, paying for the full reserved duration regardless of actual usage. Fits a planned, time-critical event where "as soon as available" isn't good enough — a product launch, a known seasonal peak.

Meridian's actual workloads sit mostly in the bottom-left — commodity Compute Engine and Cloud Run capacity with ordinary quota headroom is enough for everything this course's throughline runs. DWS becomes relevant the day Meridian's data team requests GPU capacity for a route-optimization model retraining job — the scarce, flexible-timing case Flex-Start mode fits exactly.

Forecasting From Real Data With PromQL, Not Guesswork#

Capacity Planning & Performance's forecasting section already taught the method — pull real historical demand data, don't rely purely on trend extrapolation, plan against peak rather than average. The GCP-specific piece: Cloud Monitoring now natively accepts PromQL queries against its own metric data, including in the Metrics Explorer and alerting policies — the same query language GCP Cloud Engineer Foundations Part 8 introduced for Managed Service for Prometheus, usable against any Cloud Monitoring metric, not just Prometheus-sourced ones. MQL, Cloud Monitoring's older text-based query language, still works for existing dashboards but is no longer the recommended path for new work — PromQL is.

# Meridian's actual capacity-forecasting query ahead of a known seasonal
# peak: the 95th-percentile CPU utilization across gps-ingestion's MIG,
# over each of the past 12 weeks, to see the real trend rather than
# guess from the last few days alone
quantile_over_time(0.95,
  compute_googleapis_com:instance_cpu_utilization{
    resource.type="gce_instance",
    metadata.user_labels.app="gps-ingestion"
  }[12w:1d]
)

This is the query Priya's team now runs before every seasonal peak (the same weather-driven demand pattern that caused this chapter's opening incident recurs, less severely, every year around the same season) — a real 12-week p95 trend, not a felt sense of "traffic's been growing," is what actually justifies a pre-emptive MIG size floor or a Cloud Run min-instances bump ahead of the next predictable spike, closing the loop between this chapter's reactive incident-mitigation levers and genuinely proactive capacity planning.

Where Autoscaling Already Lives in This Site — A Deliberate Recap#

This section is intentionally short, on purpose: MIG autoscaling policies (metric-based, predictive, scheduled) are covered in depth in GCP Cloud Engineer Foundations Part 4, GKE's HPA/VPA/Cluster Autoscaler and Cloud Run's concurrency/instance model in Part 5, and the underlying scaling formula, cold-start problem, and load-testing methodology in Capacity Planning & Performance Part 3. Repeating any of that here would be exactly the duplication this course's scoping deliberately avoids.

Mitigating Impact, Lever 1: Draining and Redirecting Traffic#

Draining traffic away from an unhealthy backend is almost always the fastest available mitigation, because it doesn't require provisioning anything new — it just stops sending requests to the thing that's failing. GCP exposes this mechanism differently depending on which compute platform is serving the traffic.

Compute Engine behind an external/internal Application Load Balancer — connection draining on the backend service delays removing an instance from rotation until its in-flight connections finish (or the timeout elapses), rather than cutting them off mid-request:

# Set a 60-second connection-draining window on shipment-api's backend
# service — long enough for an in-flight tracking request to finish,
# short enough not to stall an emergency instance removal
gcloud compute backend-services update shipment-api-backend \
  --global \
  --connection-draining-timeout=60

Cloud Run — traffic is already split by percentage across revisions; redirecting is an immediate update-traffic call, not a drain-and-wait:

# Immediately send 100% of traffic back to the last known-good
# revision — the fastest traffic-redirect GCP offers, since Cloud
# Run has no connection-draining wait built into this path at all
gcloud run services update-traffic shipment-api \
  --project=meridian-shipment-prod \
  --to-revisions=shipment-api-00042-abc=100

GKE behind Gateway API — an HTTPRoute's backendRefs weights redirect traffic between Services with the same mechanism used for canary rollouts, just pointed in the opposite direction:

# Redirect 100% of route-optimizer traffic away from a suspect new
# version back to the previous stable Service, by setting weights
# to the extremes rather than a canary's typical 95/5 split
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: route-optimizer-route
spec:
  rules:
    - backendRefs:
        - name: route-optimizer-stable
          port: 80
          weight: 100
        - name: route-optimizer-canary
          port: 80
          weight: 0

What to notice: every platform's mechanism does the same two things in the same order — stop new traffic, let existing traffic finish — the only real difference is how fast that happens (Cloud Run's traffic-split update is near-instant; a Compute Engine connection drain waits out its configured timeout by design).

Mitigating Impact, Lever 2: Adding Capacity Fast#

Draining traffic helps when the fix is "stop sending requests to the broken thing." Adding capacity helps when the actual problem is "there isn't enough of the healthy thing." GPS-ingestion's weather-event spike from this chapter's opening story is squarely this second case — nothing was unhealthy, there just wasn't enough Cloud Run consumer capacity to keep up with 4x normal volume.

# The fix Devon eventually found by trial and error — raised
# deliberately here as documented policy instead: bump Cloud Run's
# max-instances ceiling immediately, ahead of any autoscaler reaction
gcloud run services update gps-ingestion-consumer \
  --project=meridian-shipment-prod \
  --max-instances=200 \
  --concurrency=40

# The equivalent manual override for a Compute Engine MIG — bypasses
# the autoscaler's own target-tracking delay by setting size directly
gcloud compute instance-groups managed resize gps-ingestion-mig \
  --size=40 \
  --zone=us-central1-a

🔍 From the Trenches: a different team at a company migrating a similar workload to GKE configured their overflow node pool entirely on Spot VMs to save cost, and during a real traffic-spike incident, the overflow nodes themselves got preempted nine minutes after joining the cluster — compute reclaimed by Google mid-incident, on top of the original capacity shortage. The immediate cause was choosing Spot for emergency headroom specifically because it's what the routine autoscaling config already used; the underlying condition was never distinguishing "Spot is fine for routine elastic scaling, because losing a Spot instance during normal operation is a minor, expected event" from "Spot is a real liability for emergency overflow capacity, because losing it while already degraded turns a manageable incident into a worse one." The fix was a separate, standard-VM overflow node pool reserved specifically for incident response, never touched during normal autoscaling.

Mitigating Impact, Lever 3: Rollback Strategies#

When the actual root cause is "the last deploy introduced this," rollback is the correct lever — not draining, not adding capacity, since both of those treat the symptom while the bad code keeps running. GCP DevOps & CI/CD Platform Part 4 already built Cloud Deploy's automated rollback and canary-abort mechanics; this section is the incident-response-specific application of that same capability.

# Roll back the meridian-prod target to its last known-good release —
# the single command that undoes exactly what Part 4's Cloud Deploy
# pipeline just promoted, without touching draining or capacity at all
gcloud deploy targets rollback meridian-prod \
  --delivery-pipeline=shipment-api-pipeline \
  --region=us-central1

For Cloud Run services deployed outside a full Cloud Deploy pipeline (a hotfix path, say), the traffic-split mechanism from Lever 1 doubles as an instant rollback — redirecting 100% of traffic to the previous revision is the rollback, with no separate rollback API needed. For a GKE Deployment managed outside Cloud Deploy, kubectl rollout undo deployment/route-optimizer reverts to the prior ReplicaSet using Kubernetes' own revision history — the generic mechanism, not GCP-specific, but worth naming here because it's the one gap in an otherwise Cloud-Deploy-centric rollback story.

A Real Incident Playbook: All Three Levers Together#

Most real incidents don't cleanly fit one lever — Meridian's own weather-event spike started as a pure capacity problem, but the fix Devon eventually found (raising max-instances) took several minutes to fully take effect while the backlog kept growing, and the team briefly considered draining a fraction of GPS-ingestion traffic to a lower-fidelity degraded-mode consumer while capacity caught up.

What to notice: the diagnosis step is the one this chapter's opening incident actually skipped — Devon jumped straight to guessing a fix instead of first confirming which of the three trigger conditions actually applied. Part 5's troubleshooting workflow builds the systematic version of that diagnosis step; this chapter's contribution is making sure a lever exists and is documented for each of the three outcomes that diagnosis can produce.

Choosing the Right Lever — A Decision Framework#

Terminology Map: Incident-Mitigation Levers Across AWS, Azure, and GCP#

A Full Worked Example: Meridian's Incident Response Runbook#

The concrete artifact Meridian's team wrote after the weather-event incident — the decision tree from earlier in this chapter, turned into an actual on-call runbook entry:

# Meridian's GPS-ingestion incident runbook, condensed to command form
# — the exact sequence the team now follows instead of guessing

# 1. Confirm which trigger condition applies (diagnosis, per this
#    chapter's playbook state diagram)
gcloud logging read \
  'resource.type="cloud_run_revision" resource.labels.service_name="gps-ingestion-consumer" severity>=WARNING' \
  --project=meridian-shipment-prod --limit=50 --freshness=15m

# 2a. If it's a capacity shortfall (this incident's actual cause):
gcloud run services update gps-ingestion-consumer \
  --project=meridian-shipment-prod --max-instances=200

# 2b. If it's a bad deploy instead:
gcloud deploy targets rollback meridian-prod \
  --delivery-pipeline=gps-ingestion-pipeline --region=us-central1

# 2c. If it's one unhealthy backend among many:
gcloud compute backend-services update gps-ingestion-backend \
  --global --connection-draining-timeout=60

# 3. Verify: burn rate back under threshold within 10 minutes
gcloud monitoring slo describe gps-ingestion-availability \
  --service=gps-ingestion --project=meridian-shipment-prod

🧪 Hands-on checkpoint: run step 1's log query against a real (or practice) Cloud Run service under load and confirm you can distinguish a capacity-shortfall signature (rising request latency, no new deploy in the recent Cloud Build history) from a bad-deploy signature (errors starting at a timestamp matching a specific gcloud deploy rollout) before reaching for step 2 — the diagnosis step this chapter's opening incident skipped is exactly what this checkpoint exercises.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: shipment-api's error rate spikes at the same timestamp a new release finished deploying to meridian-prod, and separately, overall request volume is at a normal Tuesday-afternoon baseline. Which lever should Devon reach for, and which two levers should he specifically not reach for?

Answer: Rollback (Lever 3) — the timing correlation with a specific release, combined with normal (not elevated) traffic volume, rules out a capacity shortfall. He should not add capacity (there's no volume problem to add capacity for, and more instances just run the bad code faster) and should not drain a specific backend (the problem is the deployed code itself, not one unhealthy instance among healthy ones) — gcloud deploy targets rollback is the lever that actually addresses the root cause.

Problem 2: During an active incident, a responder wants to add overflow capacity to a GKE node pool and proposes using Spot VMs "since it's faster to provision and cheaper." What's the risk this chapter specifically warns about, and what should the responder do instead?

Answer: A Spot VM added as emergency overflow capacity can be preempted mid-incident, turning an already-degraded system into a worse one exactly when it can least afford a further capacity loss — the "From the Trenches" example in this chapter shows this happening nine minutes after the overflow nodes joined the cluster. The responder should provision standard (non-Spot) overflow capacity for incident response specifically, accepting the higher cost as the correct tradeoff during an active outage, reserving Spot for routine, fault-tolerant elastic scaling instead.

Problem 3: A team wants to use Dynamic Workload Scheduler for their everyday Cloud Run autoscaling, reasoning that "reserved capacity sounds more reliable than on-demand." Is this the right tool, and why or why not?

Answer: No — DWS exists specifically for scarce, high-demand resources (GPUs, TPUs, capacity-constrained regions) where on-demand provisioning isn't guaranteed to succeed even with sufficient quota. Cloud Run's everyday autoscaling draws from commodity, broadly available compute capacity, where standard on-demand provisioning plus adequate quota headroom (and the Quota Adjuster to keep that headroom current) is the correct, simpler mechanism — reaching for DWS here adds real complexity (Calendar mode's upfront payment for reserved duration, or Flex-Start's bounded-duration model) with no actual capacity-availability problem it's solving.

Summary and What's Next#

This chapter closed the gap Part 1's SLO/error-budget signal opened but didn't answer on its own: what to actually do once that signal fires. Quotas, limits, and reservations set the ceiling any capacity plan operates under; Dynamic Workload Scheduler reserves scarce capacity ahead of time for the cases on-demand provisioning can't guarantee; and the three incident-mitigation levers — draining traffic, adding capacity, and rollback — each map to a specific, diagnosable trigger condition rather than being interchangeable guesses. Meridian's weather-event incident, which once cost twelve minutes of confused trial and error, now has a documented runbook that gets a responder to the right lever in the time it takes to run one diagnostic query.

Part 3 shifts from reacting to incidents toward seeing them coming sooner: instrumenting telemetry properly in the first place, with the Ops Agent, OpenTelemetry, Google Cloud Managed Service for Prometheus, and Cloud Service Mesh's automatic telemetry — the raw material every alert, every SLO, and every lever in this chapter ultimately depends on actually existing and being accurate.