basicSli is a small set of pre-built indicators (availability, latency) Cloud Monitoring computes automatically for GKE/App Engine/Cloud Run services with almost no configuration. requestBased is a custom SLI expressed as a ratio of good to total requests (goodTotalRatio) or a latency-threshold cut (distributionCut) against your own metrics. windowsBased instead evaluates whether each fixed time window met a goodness criterion, useful for services where "percentage of good requests" isn't the natural unit, like a batch pipeline.

It's a Cloud Monitoring filter that returns an SLO's error-budget burn rate as a time series any alerting policy condition can evaluate. It takes the target SLO's full resource name and a lookback period (e.g., "3600s") — the burn rate is computed over that lookback window.

A single short lookback window alone fires on transient, self-resolving blips (like a brief Cloud Run cold-start 503); a single long lookback window alone reacts too slowly to a genuinely fast-burning incident. Pairing a short and long window, AND-combined, requires both to agree before paging — responsive to real incidents, resilient to noise.

A Cloud Service Mesh SLO is derived automatically from the Envoy sidecar's own request telemetry — no application code changes needed, but it only sees network-facing behavior (HTTP status, latency), not application correctness. A Service Monitoring SLO is built from metrics you define yourself, and can capture failure modes a sidecar can't see, like a request that returns a 200 with a logically wrong result.

Meridian has no ability to make Google Cloud's own infrastructure more available than Google itself contracts to deliver — composite availability across a request path is the product of each stage's uptime, not the maximum of them, so a service built on components rated at 99.95% can't honestly promise higher than that (and in practice slightly less, once composed across multiple stages).

No. The frozen tier exists to stop new risk while the team recovers from a reliability breach — an unrelated change, however finished, is exactly the kind of new risk the tier is meant to block. The override mechanism exists for the incident fix itself, not for other ready work, or the frozen tier's actual purpose quietly erodes.

Add a second, longer-lookback condition (e.g., 1 hour) AND-combined with the short one, so both must exceed the threshold before paging. The single-window version failed because a brief blip alone was enough to trip it — there was no requirement that the burn also be sustained over a longer period, which is exactly what separates a real incident from noise.

The mesh-derived SLO is computed from sidecar telemetry — it sees the request succeed with a 200 response regardless of whether the returned route is actually correct. A wrong-but-successfully-returned response is invisible to that SLI by construction; catching it requires a second, application-level SLI that specifically validates response correctness.

Price the specific GCP architecture change that target actually requires (typically multi-region active-active infrastructure, automated failover, dedicated on-call) rather than treating "more nines" as a free request, and confirm the business case still holds once the real infrastructure and operational cost is quoted.

Verify zero real consumers over a full business cycle, not just a quiet observation window — an idle-looking resource (low CPU, no recent deploys) is not the same as confirmed zero downstream dependents. Meridian's own legacy-invoice-sync retirement found an undocumented nightly job still consuming a file it wrote, discovered only because the team watched for consumers rather than assuming none existed.

A quota is a per-project or per-region cap GCP enforces on a resource, and can typically be raised by request. A limit is a hard, non-negotiable system limit that can't be raised at all. A reservation is pre-purchased, guaranteed capacity for a specific machine type/zone that matching VMs consume automatically — it guarantees availability, not just permission to request it.

Even an automatically-approved quota increase typically takes minutes to process, and one requiring manual Google review can take days — neither is fast enough to help during an active incident. Quota headroom has to already exist before the incident starts; requesting more mid-incident helps the next incident, not the current one.

It observes real usage trends and proactively submits increase requests automatically before a project gets close to its ceiling, rather than waiting for a human to notice during a capacity review or an incident — turning quota management from reactive to proactive.

Flex-Start mode fits a workload with a flexible start time but a real deadline — it requests capacity for a bounded duration (1 minute to 7 days), fulfilled as soon as available, billed only for actual usage. Calendar mode fits a planned, time-critical event where "as soon as available" isn't good enough — it reserves capacity for a specific future window up to 90 days out, paid for the full reserved duration regardless of usage.

Draining/redirecting traffic addresses a localized unhealthy backend. Adding capacity addresses a genuine demand spike with no recent deploy. Rollback addresses an error signature that correlates with a specific recent release.

Rollback fits — the timing correlation with a release, combined with normal traffic, points at the deployed code itself. Adding capacity doesn't fit (there's no volume problem, and more instances just run the bad code faster); draining a specific backend doesn't fit either (the problem is the deployed code, not one unhealthy instance among healthy ones).

A Spot VM can be preempted by Google mid-incident, turning an already-degraded system into a worse one at the worst possible time. The responder should provision standard (non-Spot) capacity for incident-response overflow specifically, reserving Spot for routine, fault-tolerant elastic scaling where a preemption is a minor, expected event rather than a compounding failure.

The postmortem can't determine which change actually fixed the issue, since both were applied together — a real cost when the same failure mode recurs later and the team needs to know immediately which lever to reach for, rather than repeating the same ambiguous double-fix.

Because it re-runs the entire delivery pipeline for the prior release, including whatever approval gate that release originally required — an unreviewed emergency rollback would itself be a real risk, so the gate stays in place. A responder should know in advance whether the target has an approval gate and factor that approval time into incident response time.

Query real historical utilization data directly — a PromQL query against Cloud Monitoring's own metrics (e.g., quantile_over_time for a 95th-percentile CPU trend over the past several weeks) gives an actual trend to plan against, rather than an intuition-based guess, matching the "forecast from real data" discipline Capacity Planning & Performance already taught generically.

A metrics collector (built on the OpenTelemetry Collector) and a logging collector (built on Fluent Bit), both controlled by one YAML config file. The metrics side collects infrastructure metrics; the logging side tails and ships log files/syslog.

Agent/sidecar runs one collector per host or pod — more collectors to manage and more base resource overhead, but no shared point of failure. Gateway runs one shared collector per cluster/region — centralizes processing like redaction or sampling in one place, but an outage of that single collector affects every workload routed through it unless it's run with real availability discipline (multiple replicas, a PodDisruptionBudget).

A full exclusion filter drops every matching entry before storage — appropriate for genuinely zero-value noise like health-check logs. A sampling exclusion filter (using sample()) keeps a configured percentage and drops the rest — appropriate for high-volume but not zero-value logs, like routine 2xx load-balancer traffic, where some representative fraction still has analytical value.

Managed collection runs Google-operated collectors reading your PodMonitoring/ClusterPodMonitoring resources — lowest operational burden. Self-deployed collection runs your own Prometheus server or OTel Collector configured to remote-write into the same managed storage backend — for teams with an existing Prometheus scrape-config investment who still want Google's storage/query layer.

Every self-deployed collector already writes into the same shared global backend (Monarch) — a single query or a GlobalRules recording rule can already aggregate across every cluster and project directly, so a federation layer solves a problem this architecture doesn't actually have, adding real operational complexity for no benefit.

A silent telemetry coverage gap — a subset of nodes lost sidecar injection during the upgrade, so a fraction of real traffic never emitted the telemetry the SLO's SLI depended on. The SLO is computing an accurate number from an incomplete sample, not a wrong number — the fix is verifying sidecar-proxy container counts match pod counts directly, rather than trusting the SLO's healthy output as proof telemetry itself is complete.

Every workload routed through that one collector silently drops telemetry for the outage's full duration — not just the drained node, since a gateway centralizes collection for potentially many services at once. This is why the collector itself needs the same availability discipline (multiple replicas, a PodDisruptionBudget) as the production services it observes.

In the short term, a distribution-type log-based metric with a value-extractor pulling the latency field out of each matching log entry — this backs real percentile queries the way a counter-only log-based metric can't. The better long-term fix is adding a native OpenTelemetry histogram metric directly in the application, since log-based metrics carry real ongoing parsing cost on every matching log entry that a native metric doesn't.

A synthetic monitor — it's active, running its own scripted probe on a schedule regardless of real traffic, unlike every other mechanism in this chapter (Ops Agent, mesh telemetry, application metrics), which are all passive and only surface a problem if real traffic happens to hit the broken path during the gap.

Infrastructure telemetry (Ops Agent) and mesh telemetry first, since both require zero application code changes and give an immediate baseline; application-level OpenTelemetry instrumentation next, for the metrics only the application itself knows about; a synthetic monitor last, for the specific workflows where a real-traffic gap would be unacceptable to miss. The order follows cost and specificity — cheapest and most general first, most deliberate and narrowly-scoped last.

A Log Analytics linked dataset is a live view over the log bucket's own data — no separate copy, no ingestion/storage cost beyond the bucket itself. An export sink to BigQuery creates an actual separate copy of the data in a BigQuery table, with its own ingestion and storage cost.

_Required holds Admin Activity and System Event audit logs at a fixed 400-day retention that cannot be modified. _Default holds everything else at a 30-day default retention, configurable between 1 and 3650 days.

A log that reaches _Default has already been ingested unredacted — the whole point of routing through a Pub/Sub sink into a Dataflow pipeline calling Sensitive Data Protection (Cloud DLP) is intercepting and de-identifying the data before it's ever stored in a queryable bucket, not cleaning it up afterward.

Tokenization and format-preserving encryption are reversible (with the encryption key); masking and bucketing are irreversible. The choice depends on whether there's a genuine, audited downstream need to ever recover the real value — reversible methods fit that need, irreversible methods fit a field with no legitimate reason to ever see the real value again.

All four route through a single shared Google-internal delivery service — a failure in that shared service affects all of them simultaneously, even though they appear to be separate integrations configured independently.

The likely cause is an outage in the shared Google-internal delivery backend that PagerDuty, webhook, Slack, and Cloud Mobile App channels all route through — not a misconfigured policy. The fix is adding a genuinely independent-backend redundant channel (email or Pub/Sub) alongside the primary consolidated channel specifically for high-stakes alerts.

Increase the _Default bucket's retention to 365 days via gcloud logging buckets update. If retention is ever shortened later, logs older than the new period become immediately unqueryable (with only a 7-day grace period to undo it by raising retention again) — shortening should be a deliberate, compliance-reviewed decision, not a routine cost-cutting move.

Logs Explorer and LQL are built for filtered, targeted lookups, not multi-field aggregation and correlation across a whole dataset. Log Analytics' linked BigQuery dataset, queried with real SQL (a GROUP BY with computed ratios per segment), is the right tool for this class of analytical question.

A budget alert only fires once cumulative spend crosses a threshold — six hours of acceleration may not yet have pushed the cumulative total past that threshold, even though the rate of spend is clearly abnormal. A separate Cloud Monitoring alert on the billing metric's rate of change (e.g., daily spend 3x above a 7-day average) catches the acceleration itself, independent of the cumulative budget check.

Console-built dashboards are undocumented and unreviewed — easy for engineers to silently duplicate effort, let drift from what the team actually needs, or lose when the one engineer who built it leaves. The fix is managing dashboards as Terraform (google_monitoring_dashboard), reviewed in the same pull request as the service's own infrastructure changes, the same discipline this course has applied to every other piece of Meridian's infrastructure.

The longest span isn't necessarily on the critical path if it runs concurrently with other work — the critical path is the sequential chain of spans that actually determines total request time. A concurrent span, however long, contributes nothing to total latency if it finishes before the longest sequential chain does.

The W3C traceparent header (or GCP's legacy X-Cloud-Trace-Context) carries the trace ID and parent span ID forward on outbound calls. A service that receives it but doesn't forward it on its own outbound calls silently breaks the chain — every span downstream starts a new, disconnected trace, with no error or warning to indicate propagation failed.

logging.googleapis.com/trace (formatted as projects/PROJECT_ID/traces/TRACE_ID) and logging.googleapis.com/spanId.

Head-based sampling decides whether to keep a trace before the request runs, blind to outcome — a rare failure has the same small chance of being sampled as any routine request. Tail-based sampling decides after the full trace is available, so it can guarantee every error or genuinely slow trace is kept regardless of the overall sample rate, at the cost of buffering complete traces before deciding.

An exemplar is a specific trace ID attached to one observed data point inside a metric's distribution, letting a chart click jump directly to the trace behind that exact point. Only histograms can carry one because an exemplar attaches to an individual observation within a distribution — a counter metric has no distribution, just a running total, so there's no individual data point to attach a trace ID to.

Pull a trace waterfall for a currently-slow request — metrics already confirmed that something is slow, and the absence of errors rules out a simple log search finding a stack trace, so the trace waterfall is the tool that can show where specifically the time is going, which neither metrics nor logs alone can answer.

The mistake was treating "more sampling volume" as strictly safer without distinguishing volume from relevance — a much larger set of mostly-routine traces made the UI slower to surface the one trace that actually mattered. Tail-based sampling at a much lower overall rate, biased toward keeping errors and high-latency traces specifically, gives better diagnostic coverage at a fraction of the cost and query overhead.

The OpenTelemetry SDK's automatic instrumentation only handles propagation for supported client libraries — a raw socket call needs the traceparent header propagated by hand, and if that's missed, the downstream service's spans silently start a new, disconnected trace instead of continuing the original one. It goes unnoticed because there's no error — the waterfall just quietly stops showing anything past that point, which looks identical to "no interesting spans downstream."

Request-count is a counter, not a histogram — it has no distribution of individual observed values, only a running total, so there's no single data point within it to attach a trace ID to. Exemplars only work on histogram-type metrics, like a latency distribution, where each bucket represents real individual observations.

As of April 2026, creating, running, and editing an Investigation requires a Premium Support contract or account-team-granted access — a team without that access has no working fallback if the runbook's primary step assumes Investigations is available. The manual, systematic five-category troubleshooting workflow should be the actual default, with Investigations treated as an acceleration layer only where access is confirmed to exist.

A CPU flame graph's bar width represents time spent executing — it finds the function actually burning CPU cycles. A heap allocation flame graph's bar width represents bytes allocated during the profiling window regardless of whether the objects are still alive — it finds a function creating excessive temporary objects and generating garbage-collection pressure, a different problem from a genuine memory leak.

A trace waterfall shows where time goes across a request's path through multiple services, treating each service as a black box. A CPU flame graph shows where time goes inside one application's own code, function by function — it can identify a specific function consuming a given percentage of CPU time, a level of detail no trace span alone provides.

Inform (billing export, dashboards, cost attribution via labels — establishing visibility), Optimize (CUD purchases, Recommender-driven rightsizing, Spot strategy — active waste reduction), and Operate (budgets, labeling enforcement, a recurring review cadence — ongoing governance).

Each unique label value combination becomes its own permanently stored time series — grouping by something like a raw device ID or request ID with tens of thousands of distinct values creates tens of thousands of stored series, none of which existed before the rule was added, with no obvious single moment where the cost jump would be noticed without a deliberate review.

Autopilot bills per pod resource request and rewards accurate requests; Standard bills per node regardless of pod density and rewards active, deliberate bin-packing. A team that won't actively bin-pack is usually cheaper on Autopilot; a dense fleet of many small microservices that will be actively bin-packed (especially with Spot node pools) can be dramatically cheaper on Standard.

The likely cause is a recording rule or query grouping by a high-cardinality label added for a legitimate but narrow debugging purpose (in Meridian's real case, grouping by a 40,000-value device ID field), silently multiplying stored time series. A regular FinOps "Inform" review, tracking cost by label/service trend over time, would have surfaced the unexplained cost jump within the first review cycle after the change, rather than two months later.

Over half the committed capacity goes unused once the migration completes, while the CUD continues billing at the full committed rate for the remaining commitment term regardless of actual usage — a fixed cost paid for capacity that no longer exists. A shorter commitment term (1-year instead of 3-year), reviewed and re-committed at renewal once the migration's real trajectory is known, captures most of the same discount with far less stranded-commitment risk.

Instance-based billing — Cloud Run's default is request-based billing (no charge while idle), which is optimized for bursty, sporadic traffic. For steady, high-volume traffic, instance-based billing's lower per-second rate wins out despite billing for the instance's full lifecycle, since there's little idle time being paid for unnecessarily in the first place.

The customer-facing tracking API should stay on Premium — its lower latency (Google's private backbone) is worth the higher per-GB egress cost for a latency-sensitive, customer-facing path. The nightly BigQuery export job should use Standard — it's not latency-sensitive, and Standard's notably lower per-GB egress cost (roughly 30-60% less) is pure savings with no real downside for a bulk, non-interactive transfer.

They're missing four other value categories — security, performance, reliability, and manageability — all served by the same underlying Recommender API, just via a different --recommender type in the same gcloud recommender recommendations list command. A performance-category recommender, for instance, can surface a Compute Engine instance pegged near 100% CPU, a signal just as actionable as an idle-resource cost recommendation but answering a completely different question.