SRE Fundamentals & Reliability — Fundamentals

An SLI (Service Level Indicator) is a directly measured metric — the actual percentage of requests that succeeded in the last window, or the actual p99 latency. An SLO (Service Level Objective) is the internal target you hold that SLI to — "99.9% of requests succeed over a rolling 30 days." An SLA (Service Level Agreement) is the external, usually contractual, promise to customers, almost always set looser than the SLO — for example a 99.9% SLO backing a 99.5% SLA — so that normal SLO-driven engineering responses (freezing risky launches, prioritizing a reliability fix) kick in well before you'd ever be in breach of an actual customer contract with financial penalties attached. The distinction matters because teams that only track the SLA number find out about a reliability problem at the same moment a customer starts asking for a service credit; teams that track the SLI against the SLO get an internal early-warning system instead.

Toil is operational work that's manual, repetitive, automatable, tactical (reactive, not strategic), and — critically — has no enduring value: doing it once doesn't make the system better, and it scales linearly with traffic or team size rather than staying flat. A manual database failover you run once a quarter isn't automatically toil just because it's manual; it's toil specifically if it keeps recurring and adds nothing durable each time. To reduce it on a team that's never measured it, I wouldn't start with automation — I'd start with a two-week time audit where the on-call rotation logs every task and roughly how long it took, categorized as toil vs engineering work vs overhead. That gives you a number (Google's own target is toil under 50% of an SRE's time) and, more usefully, a ranked list. From there I'd automate the highest-frequency, lowest-complexity item first to build momentum and prove the approach, rather than picking the most annoying one, which is often also the hardest to automate safely.

(a sample shape to adapt to your real background, not a script to memorize) — "At my last role I owned the SLO for our order-processing API: 99.95% success rate on a rolling 28-day window, measured off request-level success/failure counters we already emitted, with the error budget tracked on a Grafana panel the whole team could see. The most concrete thing it drove: three weeks before a major feature launch, we were already at 60% of that month's error budget burned from an unrelated dependency issue. I used that number to make the case — successfully — for delaying the launch by a week rather than shipping into an already-degraded reliability posture, which is a conversation that's much easier to win with a specific number than with a vague 'things feel shaky right now.'"

First confirm the number itself and its trend — is the burn from one incident that's already resolved, or an ongoing elevated error rate that will keep consuming budget through the two launches? Those require different responses. Check whether an error budget policy already exists for this service (a pre-agreed rule like "below 20% remaining budget, only reliability work and rollback-safe changes ship") — if it does, the job is applying it, not re-litigating it. If it doesn't exist, this incident is itself the argument for writing one going forward. Bring the burn-rate trend and the specific launches to the stakeholders together, and separate the three launches by actual risk — a config flag flip is not the same risk as a new payment code path — rather than treating "pause everything" or "ship everything" as the only two options.

"First I'd pull the actual burn-rate trend, not just the current 90% number — if it's a single resolved incident, the remaining budget is a known, fixed quantity and the conversation is different than if the error rate is still elevated right now, which means budget keeps burning through both launch windows. I'd check if we already have an error budget policy for this service; if we do, I'm applying it, not re-negotiating it in the moment — that's the entire point of agreeing to the policy in advance, before anyone has a launch riding on the outcome. If we don't have one, I'd say so explicitly and treat that gap as part of the postmortem for whatever caused this burn. Then I'd separate the three launches by actual blast radius — I'd push to delay the one touching the payment path and let a low-risk config-only change through, rather than a blanket freeze that makes every future budget conversation harder to have credibly because it looks like the rule only gets enforced when it's convenient."

"Two issues. First, the denominator is wrong — the rule divides the current error ratio by 0.999, the SLO target itself, but the burn-rate formula divides by the allowed error fraction, which is 1 - SLO, or 0.001 here. Dividing by 0.999 instead of 0.001 makes the computed burn rate off by almost a thousand-fold, so the actual error ratio would need to be around 1400% — mathematically impossible — to ever cross the 14.4 threshold. As written, this alert can functionally never fire; it looks like working alerting but it's silently dead. Second, and just as important: this is a single-window alert even though it's named 'Fast' as if it's part of a multi-window strategy. Google's SRE workbook approach pairs a short confirming window (like 5 minutes) with the longer window (1 hour here) so the alert requires both to be elevated before paging — that's what filters out a brief blip from a real, ongoing burn. A lone 1-hour window will still page on transient spikes that have already resolved by the time anyone looks at it, and it lacks the fast-detection half of what a 'fast burn' alert is supposed to provide in the first place."

What's the current baseline reliability, and is there an existing SLA with customers or merchants that this SLO needs to sit comfortably inside? Is checkout synchronous end-to-end, or does it hand off to an async payment-processing step — because that changes what "success" even means for the request-scoped SLI. What's the traffic pattern — is there a hard peak (e.g., a flash sale) where the same SLO target implies a very different absolute error budget in requests? Who's the on-call audience this alerting needs to serve — a dedicated platform SRE team, or the feature team that owns checkout directly?

Pick a small number of user-facing SLIs rather than a metric per component — request success rate (5xx and clearly-broken 2xx responses, like an empty cart total, both count as failures) and p99 latency, since a checkout that "succeeds" after an unacceptable delay is a failure from the user's perspective even if it returns 200. Set the SLO slightly tighter than any external SLA (say 99.95% success over a rolling 30 days, backing a looser 99.9% external commitment) so the error budget gives real reaction room. Build multi-window, multi-burn-rate alerting on top: a fast pair (5-minute and 1-hour windows, ~14.4x burn rate) that pages immediately for an acute spike, and a slow pair (6-hour and 3-day windows, ~6x and 1x burn rate) that files a ticket rather than paging for a slow, sustained leak that a human should look at during business hours, not at 3am. Publish the error budget on a dashboard the checkout team actually looks at, tied to a pre-agreed error budget policy for what happens as it depletes.

A single tight SLO everywhere is simple to reason about but ignores that checkout during a flash sale has a much larger absolute request volume — the same 99.95% target represents far more failing requests in absolute terms, which may warrant a temporarily stricter target or extra capacity headroom rather than treating the SLO as a flat, context-free number. Paging on the fast burn-rate pair catches real acute incidents quickly but will still produce occasional false pages on a true short, self-resolving blip; that's a deliberate trade against the alternative of a slower, quieter alert that would miss a real outage for the first hour. I'd also flag that treating "success rate" as the only SLI risks missing a systemic slow degradation that never technically errors — which is exactly why latency needs to be its own SLI, not a side note on the success-rate one.

"I'd track two SLIs — request success rate and p99 latency, both defined at the API boundary so they reflect what the user actually experiences — against a 99.95% success SLO over a rolling 30-day window, deliberately tighter than whatever external SLA exists. On top of that I'd build the standard multi-window, multi-burn-rate alerting: a 5-minute-and-1-hour fast pair that pages on-call for an acute burn, and a 6-hour-and-3-day slow pair that opens a ticket instead of paging, since a slow leak doesn't need someone woken up for it. I'd make sure the error budget is visible on a dashboard the checkout team actually watches, not buried in an SRE-only tool, and that there's a written policy for what happens as it depletes — otherwise the whole system just becomes a number nobody acts on until it's already a crisis. The one thing I'd push to get an answer on before finalizing this: whether flash-sale traffic needs its own capacity and alerting posture, because the same percentage SLO hides a much bigger absolute-error problem at 10x normal volume."