SLIs, SLOs, Error Budgets & Service Lifecycle

Assumes you're comfortable with GCP's resource hierarchy, IAM, and CI/CD platform design at the level covered in GCP Cloud Engineer Foundations and GCP DevOps & CI/CD Platform, and with SLI/SLO/error-budget theory at the level covered in SRE Fundamentals — this course builds the GCP-specific operating layer on top of both rather than re-teaching either.

Table of Contents#

1. Why This Course Exists — PCDE Sections 3-5
2. Meridian Logistics: From Shipping Pipelines to Operating Them
3. What This Course Does Not Re-Teach
4. Cloud Monitoring's Service Monitoring: Turning an SLO Into a GCP Object
5. Defining Meridian's First SLO: shipment-api Availability
6. Multi-Window, Multi-Burn-Rate Alerting on GCP
7. Wiring the Error Budget Policy Into Automation
8. Error Budgets for Service-Mesh Workloads
9. The Opportunity Cost of "Nines" — Priced in GCP Dollars
10. Where GCP's Own SLAs Set Meridian's Ceiling
11. Managing Service Lifecycle: Planning Through Retirement
12. Terminology Map: SLO Tooling Across AWS, Azure, and GCP
13. A Full Worked Example: Meridian's SLO-to-Release-Gate Pipeline
14. Common Mistakes and Interview Traps
15. Worked Practice Problems
16. Summary and What's Next

Why This Course Exists — PCDE Sections 3-5#

This course covers the operational two-thirds of the Professional Cloud DevOps Engineer exam that GCP DevOps & CI/CD Platform didn't: applying SRE practices (~18%), observability and troubleshooting (~25%), and performance/cost optimization (~12%) — a combined 55% of the exam, and the largest single block of it. Where that course built the pipeline that gets code into production safely, this course builds the discipline and tooling that keeps it running well once it's there, and tells you fast and precisely when it isn't.

Google's own exam guide is specific about what "applying SRE practices" means for PCDE: balancing change velocity and reliability through SLIs, SLOs, SLAs, and error budgets; managing a service's full lifecycle including capacity planning and autoscaling; and mitigating incident impact through traffic draining, added capacity, and rollback. This chapter covers the first of those three — not by re-deriving what an SLO is or how error-budget math works (that's already a complete, dedicated topic on this site — see the next section), but by showing exactly how Cloud Monitoring turns an already-designed SLO into a live GCP object with real alerting behavior wired to it.

Meridian Logistics: From Shipping Pipelines to Operating Them#

If you've worked through the first two courses in this series, you know Meridian Logistics — the freight-tracking company whose platform team (Priya, Devon, and Ana) bootstrapped a GCP landing zone, then built a full CI/CD platform on top of it: a dedicated tooling project, Terraform-managed infrastructure, Cloud Build CI, and a Cloud Deploy pipeline with canary rollouts, approval gates, and a supply-chain security gate enforced by Binary Authorization.

This course picks up two months after that pipeline went live. The pipeline itself works exactly as designed — releases move through dev, staging, and production with automated promotion, vulnerability scanning, and a human approval gate in front of meridian-prod. But Priya's team has run into a different, subtler problem: the pipeline can ship a release safely, but nothing in it knows whether that release was actually a good idea. Cloud Deploy's approval gate is a human clicking "approve" after reading a changelog, not a signal grounded in how the service is actually performing. Two releases ago, a change to shipment-api's payment-processing path passed every CI check, cleared vulnerability scanning, and got a QA sign-off — and then quietly doubled p99 latency on the checkout path for six hours before anyone noticed, because nothing downstream of "the deploy succeeded" was watching whether the service was still healthy.

That gap — a pipeline that can ship safely but can't yet reason about reliability — is what this course closes. Every chapter builds one piece of Meridian's answer to it: starting here, with turning "is shipment-api healthy" from a vague feeling into an SLO Cloud Monitoring can evaluate every minute and an error budget the Cloud Deploy pipeline can actually consult before letting the next risky release through.

What to notice: the incident that opens this course isn't a pipeline failure — Cloud Deploy did exactly what it was told to do. It's an observability failure: the pipeline had no reliability signal to consult, so it had none to act on. That distinction — a working delivery mechanism with no feedback loop attached to it — is the throughline for this entire course.

What This Course Does Not Re-Teach#

Three deep, standalone series on this site already cover the theory this course builds on. Reading them first (or alongside this course) will make every chapter here land faster, because this course assumes the concept and teaches only the GCP-specific mechanism:

A skimming reader who has already read those series should notice this course reusing their vocabulary constantly (SLI, error budget, burn rate, RED, golden signals) without re-explaining any of it — that's deliberate, not an oversight. GCP Cloud Engineer Foundations Part 8 already introduced Cloud Monitoring, Cloud Logging, Cloud Trace, Cloud Profiler, and Error Reporting at an Associate Cloud Engineer depth — configuring a basic alert, reading a trace waterfall. This course goes one level deeper into each of those same products at the depth PCDE actually tests: SLO objects instead of ad-hoc thresholds, multi-window burn-rate alerting instead of a single static condition, log analytics instead of a single saved query, and a systematic multi-tool troubleshooting workflow instead of reaching for one tool at a time.

Cloud Monitoring's Service Monitoring: Turning an SLO Into a GCP Object#

Service Monitoring is the Cloud Monitoring subsystem that represents an SLO as a first-class API object, not just a number in a spreadsheet or a comment on a dashboard. Once an SLO exists as a ServiceLevelObjective resource, Cloud Monitoring computes its current compliance and remaining error budget continuously, and — critically for the rest of this chapter — exposes that computed error-budget burn rate as a time series any alerting policy can evaluate, the same way it evaluates a CPU-utilization metric.

Three pieces compose an SLO object:

A Service — the thing being measured. GCP auto-discovers services for App Engine, GKE, and Cloud Run workloads; for anything else (a Compute Engine-hosted API, a Pub/Sub pipeline), you define a custom service explicitly, giving Service Monitoring a name to attach SLOs to.

A Service-Level Indicator (SLI), expressed one of two ways:

• basicSli — a small set of pre-built indicators (availability, latency, and for GKE/App Engine, a few more) that Cloud Monitoring computes for you from the service's own request metrics, with almost no configuration. Fastest to stand up, least flexible.
• requestBased / windowsBased — a custom SLI you define directly against your own metrics. requestBased computes a ratio of good requests to total requests (via a TimeSeriesRatio filter pair, or a DistributionCut for a latency threshold). windowsBased instead evaluates whether each fixed time window (say, every 5 minutes) met a goodness criterion — useful for services where "percentage of good requests" isn't the natural unit, like a batch pipeline that's either healthy or not in a given window.

What to notice: whichever SLI type you choose, the output is the same — a computed burn-rate time series. That convergence point is what Part 1's later alerting section builds on: the alerting policy syntax doesn't care whether the underlying SLI was basicSli or a hand-built distributionCut.

Defining Meridian's First SLO: shipment-api Availability#

Priya's team picked shipment-api's public tracking endpoint as the first service to formalize, because it's the one directly visible to Meridian's customers and it's the one the silent p99 regression from this chapter's opening story actually hit. The SLI already exists conceptually from SRE Fundamentals' user-journey method: "a customer's tracking request gets a successful, reasonably fast response." The new work here is expressing it as a requestBased goodTotalRatio SLI against shipment-api's real Cloud Run metrics.

# 1. Register shipment-api as a custom service, since it runs on Cloud Run
#    and Cloud Monitoring's auto-discovery already covers it — this step
#    is here for illustration; auto-discovered services skip it entirely
gcloud monitoring services create shipment-api \
  --project=meridian-shipment-prod \
  --display-name="shipment-api"

# 2. Create the SLO itself: 99.5% of requests over a rolling 28-day
#    window must be non-5xx, matching the availability target Priya's
#    team already agreed on using SRE Fundamentals' own SLO process
gcloud monitoring slo create \
  --service=shipment-api \
  --project=meridian-shipment-prod \
  --display-name="99.5% Availability, Rolling 28 Days" \
  --goal=0.995 \
  --rolling-period=28d \
  --request-based-sli \
  --total-service-filter='resource.type="cloud_run_revision" resource.label.service_name="shipment-api" metric.type="run.googleapis.com/request_count"' \
  --good-service-filter='resource.type="cloud_run_revision" resource.label.service_name="shipment-api" metric.type="run.googleapis.com/request_count" metric.label.response_code_class!="5xx"'

The same object, expressed as Terraform — the form Meridian's platform team actually keeps in version control, consistent with the Infrastructure-as-Code discipline GCP DevOps & CI/CD Platform Part 2 established:

resource "google_monitoring_slo" "shipment_api_availability" {
  service      = google_monitoring_custom_service.shipment_api.service_id
  slo_id       = "availability-28d"
  display_name = "99.5% Availability, Rolling 28 Days"

  goal            = 0.995
  rolling_period_days = 28

  request_based_sli {
    good_total_ratio {
      total_service_filter = "resource.type=\"cloud_run_revision\" resource.label.service_name=\"shipment-api\" metric.type=\"run.googleapis.com/request_count\""
      good_service_filter  = "resource.type=\"cloud_run_revision\" resource.label.service_name=\"shipment-api\" metric.type=\"run.googleapis.com/request_count\" metric.label.response_code_class!=\"5xx\""
    }
  }
}

Multi-Window, Multi-Burn-Rate Alerting on GCP#

An SLO object alone doesn't page anyone — it's a number Cloud Monitoring tracks. The alerting policy is the piece that turns "the error budget is burning fast" into a notification, and Cloud Monitoring has a purpose-built filter function for exactly this: select_slo_burn_rate, which takes the target SLO's resource name and a lookback period, and returns the burn-rate value over that lookback window as a time series any conditionThreshold can compare against.

Observability Part 3 already built Google's standard multi-window, multi-burn-rate table from first principles and explained why a short lookback window alone produces false positives from transient blips, while a long lookback window alone reacts too slowly to a real fast-burning incident — the fix being to require both a short and a long window to agree before paging. GCP's implementation of that same design is two alerting policies (or one policy with two AND-combined conditions) against the same SLO, each using a different select_slo_burn_rate lookback:

{
  "displayName": "shipment-api SLO — Fast Burn (page immediately)",
  "combiner": "AND",
  "conditions": [
    {
      "displayName": "1h window, 14.4x burn rate",
      "conditionThreshold": {
        "filter": "select_slo_burn_rate(\"projects/meridian-shipment-prod/services/shipment-api/serviceLevelObjectives/availability-28d\", \"3600s\")",
        "comparison": "COMPARISON_GT",
        "thresholdValue": 14.4,
        "duration": "0s"
      }
    },
    {
      "displayName": "5m window, 14.4x burn rate",
      "conditionThreshold": {
        "filter": "select_slo_burn_rate(\"projects/meridian-shipment-prod/services/shipment-api/serviceLevelObjectives/availability-28d\", \"300s\")",
        "comparison": "COMPARISON_GT",
        "thresholdValue": 14.4,
        "duration": "0s"
      }
    }
  ],
  "notificationChannels": ["projects/meridian-shipment-prod/notificationChannels/CHANNEL_ID"]
}

What to notice: the two conditions query the same SLO through the same select_slo_burn_rate function, differing only in the lookback duration passed as the second argument — Cloud Monitoring's own burn-rate computation does the arithmetic that Observability Part 3 showed you by hand with PromQL, so there's no separate rate-of-change calculation to write.

🔍 From the Trenches: Meridian's first attempt at this alert used a single condition with a 5-minute lookback and no long-window pair. It fired accurately during the real incident that motivated this chapter — but it also fired twice during a routine deploy where Cloud Run briefly served 503s during instance cold-start, each time for under ninety seconds. The on-call engineer started treating the alert as noisy within a week. The immediate cause was the missing long window meaning a short, self-resolving blip alone could trigger a page; the underlying condition was that nobody had actually implemented the two-condition AND pattern Observability Part 3 already documents — the team knew the theory but shipped the simpler, wrong version under time pressure, exactly the alert-fatigue failure mode GCP Cloud Engineer Foundations Part 8 warned about for plain threshold alerts, now recurring one layer up in SLO-based alerting.

Wiring the Error Budget Policy Into Automation#

SRE Fundamentals Part 3 already designed the policy — the tiered response (Healthy, Watch, Constrained, Frozen) that says what Meridian's team should do at each level of remaining error budget. What that course didn't cover, because it isn't GCP-specific, is making that policy enforced rather than trusted to memory — the actual gap that opened this chapter. Meridian's fix: a small scheduled Cloud Function that reads shipment-api's current error-budget consumption from the Service Monitoring API and writes the result to a Firestore document Cloud Deploy's approval workflow checks before allowing a promotion to meridian-prod.

# Runs on a Cloud Scheduler trigger every 15 minutes. Reads the SLO's
# current error-budget remaining fraction and writes the policy tier
# Cloud Deploy's approval step checks before promoting to production.
from google.cloud import monitoring_v3
from google.cloud import firestore

SLO_NAME = (
    "projects/meridian-shipment-prod/services/shipment-api/"
    "serviceLevelObjectives/availability-28d"
)

def check_error_budget_tier(request):
    client = monitoring_v3.ServiceMonitoringServiceClient()
    slo = client.get_service_level_objective(name=SLO_NAME)

    # This calls the same computed value select_slo_burn_rate surfaces
    # to alerting policies — here read directly via the API instead.
    remaining_fraction = get_remaining_budget_fraction(client, SLO_NAME)

    if remaining_fraction > 0.50:
        tier = "healthy"
    elif remaining_fraction > 0.25:
        tier = "watch"
    elif remaining_fraction > 0.10:
        tier = "constrained"
    else:
        tier = "frozen"

    firestore.Client().collection("release-gates").document("shipment-api").set({
        "tier": tier,
        "remaining_budget_fraction": remaining_fraction,
        "checked_at": firestore.SERVER_TIMESTAMP,
    })
    return tier

What to notice: this is the exact tier structure SRE Fundamentals Part 3 designed, now expressed as a state machine an automated check actually enforces rather than a policy document a human is trusted to remember mid-incident. meridian-cicd's Cloud Build pipeline (from GCP DevOps & CI/CD Platform Part 3) reads the same Firestore document as a pre-promotion step; in the frozen tier, the build step exits non-zero before Cloud Deploy's approval gate is even reached, converting the policy from a Slack-channel norm into a real, unbypassable control for non-emergency releases.

Error Budgets for Service-Mesh Workloads#

route-optimizer, Meridian's GKE-hosted routing service, runs behind Cloud Service Mesh (the current name for what was Anthos Service Mesh and Traffic Director) — and Cloud Service Mesh has its own, separate SLO and error-budget feature, distinct from Cloud Monitoring's Service Monitoring, worth knowing specifically because the PCDE exam guide calls it out by name. Because every request through the mesh already flows through an Envoy sidecar proxy, Cloud Service Mesh can derive request-success and latency SLIs directly from that proxy's own telemetry — no separate metric-filter configuration on the application side at all.

The sidecar proxy sees every request in and out of the pod regardless of the application's own instrumentation — which is exactly why Cloud Service Mesh can compute an SLO with zero application-code changes, at the cost of only seeing mesh-level request success/latency, not internal application logic like a database query specifically timing out.

💡 The transferable insight: a mesh-derived SLO answers "is this service's network-facing behavior healthy" cheaply and uniformly across every mesh-enrolled service, while a Service-Monitoring SLO built on application-emitted metrics (like shipment-api's payment-failure log-based metric from later in this course) can answer a narrower, more specific question a generic mesh sidecar has no visibility into. Meridian runs both, deliberately: route-optimizer's mesh SLO catches "the service is unreachable or slow" cheaply, and a second, custom Service-Monitoring SLO on its own /route-optimize endpoint's success rate catches "the service is reachable and fast but returning wrong routes," a failure mode a mesh sidecar has no way to detect since it only sees HTTP status codes, not response correctness.

The Opportunity Cost of "Nines" — Priced in GCP Dollars#

SRE Fundamentals already showed the mathematics of why each additional nine of reliability costs disproportionately more effort — the difference between 99% and 99.9% downtime-per-year is a rounding error in the SLO document but a massive difference in what's required to actually deliver it. What that generic treatment can't show is what that cost concretely looks like in GCP infrastructure spend, which is the version of this question PCDE actually expects you to reason about.

Meridian's shipment-api sits deliberately in the "worth it" quadrant at its current 99.5% target — the chart is a tool for the same conversation the CFO example above describes, not a claim that the top-right corner is always correct.

Where GCP's Own SLAs Set Meridian's Ceiling#

SRE Fundamentals already established why an internal SLO should be stricter than an external SLA — the SLO is a private early-warning line, the SLA is the contractual promise with real financial consequences for breaching it. The GCP-specific twist PCDE expects you to reason about: shipment-api's own SLA to Meridian's customers can't honestly exceed the weakest published SLA of any GCP building block sitting in its critical request path, because Meridian has no ability to make Google Cloud's own infrastructure more available than Google itself contracts to deliver.

shipment-api runs on Cloud Run in front of a Cloud SQL HA instance — its two weakest links are both rated at ≥ 99.95%. Composite availability across a request path is the product of each stage's uptime, not the minimum of them, so two independent 99.95% components in series compose to slightly below 99.95% overall (roughly 99.90%, ignoring correlated-failure effects) — which is exactly why Meridian's internal SLO target of 99.5% sits with real margin below what the underlying infrastructure alone could theoretically support, leaving room for the application's own bugs, deploy-time risk, and dependency failures the infrastructure SLA says nothing about.

Managing Service Lifecycle: Planning Through Retirement#

PCDE's "managing service lifecycle" objective covers more than the running-and-alerting phase this chapter has focused on so far — it explicitly includes planning, deployment, maintenance, and retirement, the phase most tutorials skip because it isn't as interesting to write about as launching something new. Meridian has a real example: legacy-invoice-sync, a Compute Engine-hosted batch job predating this course's entire platform, still running because nobody had a deliberate process for turning it off.

Active Assist surfaces the maintenance and retirement signals automatically, across five recommendation categories — cost, security, performance, reliability, and manageability — pulling from the same Recommender API this site's Foundations course already introduced for IAM. The category most relevant to service retirement specifically is cost: an idle-resource recommendation flagging a Compute Engine instance or a persistent disk with near-zero utilization over an extended window is often the first real signal that a service like legacy-invoice-sync has quietly become dead weight.

# List cost-category recommendations for Meridian's legacy project —
# this surfaces the idle-instance recommendation that started the
# actual retirement conversation for legacy-invoice-sync
gcloud recommender recommendations list \
  --project=meridian-legacy \
  --recommender=google.compute.instance.IdleResourceRecommender \
  --location=us-central1-a

A real retirement, once the decision is made, is a drain-verify-decommission sequence, not a delete: redirect or disable the trigger (Part 2 covers traffic draining in depth for the incident-response case, and the same underlying mechanism applies here), watch the service's own metrics confirm zero real traffic over a full business cycle (not just a quiet afternoon), archive anything with a compliance retention requirement, and only then tear down the compute and delete the Artifact Registry images. Meridian's own legacy-invoice-sync retirement took three weeks specifically because of the verification step — an undocumented downstream nightly job turned out to still depend on a file it wrote, discovered only because the team watched for consumers rather than assuming none existed.

Terminology Map: SLO Tooling Across AWS, Azure, and GCP#

A Full Worked Example: Meridian's SLO-to-Release-Gate Pipeline#

Bringing this chapter's pieces together — the actual sequence Priya's team ran to close the gap that opened this chapter:

# 1. Custom service + SLO for shipment-api (already shown above,
#    repeated here as step 1 of the full sequence)
gcloud monitoring slo create \
  --service=shipment-api --project=meridian-shipment-prod \
  --display-name="99.5% Availability, Rolling 28 Days" \
  --goal=0.995 --rolling-period=28d --request-based-sli \
  --total-service-filter='resource.type="cloud_run_revision" resource.label.service_name="shipment-api" metric.type="run.googleapis.com/request_count"' \
  --good-service-filter='resource.type="cloud_run_revision" resource.label.service_name="shipment-api" metric.type="run.googleapis.com/request_count" metric.label.response_code_class!="5xx"'

# 2. Two burn-rate alerting policies (fast + slow burn, shown earlier)
gcloud alpha monitoring policies create --policy-from-file=fast-burn-policy.json
gcloud alpha monitoring policies create --policy-from-file=slow-burn-policy.json

# 3. Cloud Scheduler job triggering the tier-check Cloud Function
#    every 15 minutes, writing the current policy tier to Firestore
gcloud scheduler jobs create http check-error-budget-tier \
  --schedule="*/15 * * * *" \
  --uri="https://us-central1-meridian-shipment-prod.cloudfunctions.net/check_error_budget_tier" \
  --http-method=POST

# 4. Cloud Build step added to the shipment-api pipeline, reading the
#    Firestore document and exiting non-zero if tier == "frozen" and
#    no override tag is present in the triggering commit message

🧪 Hands-on checkpoint: after step 3 runs at least once, query the Firestore document directly (gcloud firestore documents list --collection=release-gates --project=meridian-shipment-prod) and confirm the tier field is present and matches what gcloud monitoring slo describe reports for the same SLO's current status — if the two disagree, the tier-check function's threshold logic has drifted from the alerting policy's own thresholds, a real, avoidable inconsistency worth catching in practice before it happens in production.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: route-optimizer runs behind Cloud Service Mesh and already has a mesh-derived availability SLO showing 99.98% compliance. A customer complaint reports the service is returning routes that are technically valid but geographically nonsensical for their delivery zone. Why doesn't the existing SLO catch this, and what should Meridian's team add?

Answer: The mesh-derived SLO is computed from Envoy sidecar telemetry — HTTP status codes and latency — which sees the request succeed with a 200 response regardless of whether the route returned is actually correct. A wrong-but-successfully-returned route is invisible to that SLI by construction. The fix is a second, application-level SLI (a requestBased SLO in Service Monitoring, or a custom log-based metric checked against known-good route bounds) that specifically validates response correctness, not just response success — the same "mesh SLO answers reachability, application SLO answers correctness" split this chapter's route-optimizer example already established.

Problem 2: shipment-api's error budget just crossed into the "frozen" tier after a genuine incident, and a developer wants to ship an unrelated feature flag change they'd already finished before the incident happened. Should the release gate allow it, and why?

Answer: No — the frozen tier exists specifically to stop new risk while the team is already recovering from a reliability breach, and an unrelated feature change (however finished) is exactly the kind of new risk the policy is designed to block, regardless of how ready it is or how unrelated it feels to the developer who wrote it. The override mechanism exists for the incident fix itself, not for other work that happens to be ready at the same time — allowing "but it's unrelated and already done" through would quietly erode the frozen tier's actual purpose the same way an unvalidated alert threshold erodes trust in alerting generally.

Problem 3: A stakeholder asks Priya's team to raise shipment-api's SLO target from 99.5% to 99.95% "since it's customer-facing and reliability matters." What's the right next step before agreeing, and why?

Answer: Price the specific architecture change that target actually requires — per this chapter's reliability-cost table, 99.95% typically means multi-region active-active infrastructure, automated failover, and a dedicated on-call rotation, not just "trying harder" at the current architecture. The right next step is walking the stakeholder through what 99.95% concretely costs in GCP infrastructure spend and operational overhead, and confirming the business case still holds at that price — not accepting the higher number and discovering the true cost only once implementation starts, the same "cost of nines" conversation this chapter's CFO example resolved to 99.5% for exactly this reason.

Summary and What's Next#

This chapter turned SLI/SLO/error-budget theory — already fully covered elsewhere on this site — into working GCP mechanism: ServiceLevelObjective objects in Cloud Monitoring's Service Monitoring, multi-window burn-rate alerting policies built on select_slo_burn_rate, an error-budget policy enforced by an automated Cloud Deploy pre-promotion check instead of trusted to memory, Cloud Service Mesh's mesh-native SLO feature for route-optimizer, the real GCP-dollar cost behind each additional nine of reliability, and a deliberate process for a service's full lifecycle through retirement. Meridian's shipment-api now has a real, enforced answer to the gap that opened this chapter: a release can no longer ship cleanly through every pipeline check while the service itself is quietly unhealthy.

Part 2 goes one layer deeper into the service-lifecycle objective this chapter only introduced: capacity planning with GCP's own levers (quotas, reservations, Dynamic Workload Scheduler), autoscaling across MIGs, GKE, and Cloud Run, and the concrete incident-mitigation techniques — draining traffic, adding capacity fast, and rollback — that PCDE tests as a distinct skill from the alerting this chapter built.