SLOs and Error Budgets: Reliability Without Burnout

Your uptime target is a number nobody actually agreed on. Product wants features shipped, ops wants nothing to break, and every incident reopens the same argument with the same people. SLOs and error budgets end that argument with arithmetic: a reliability target users actually notice, and a spending account that tells you when to ship fast and when to slow down.

Why 100 percent is the wrong target

Perfect reliability is not just expensive, it is incoherent. Your users reach you through ISPs, mobile networks, DNS resolvers, and devices that fail on their own. In our experience, once your availability exceeds the reliability of the path between you and the user, additional nines are invisible to customers and brutally expensive to engineers.

The cost curve is not linear. Moving from 99 percent to 99.9 percent usually means better deploys and basic redundancy. Moving from 99.9 to 99.99 means multi-zone failover, aggressive load shedding, and an on-call rotation that actually gets paged at 3 a.m. Moving beyond that means multi-region active-active, chaos testing, and a reliability team. Each nine costs roughly an order of magnitude more than the last.

So the real question is never "how reliable can we be." It is "how unreliable can we afford to be before users care." That number is your SLO. The gap between your SLO and perfection is your error budget, and it is a feature, not a failure. It is the room you use to deploy, experiment, and run risky migrations.

Get the vocabulary straight

Three terms, constantly confused, doing very different jobs:

• SLI (Service Level Indicator): a measurement. The ratio of good events to total events over a window. Example: successful HTTP requests divided by total HTTP requests.
• SLO (Service Level Objective): an internal target for that SLI. Example: 99.9 percent of requests succeed over a rolling 30 days.
• SLA (Service Level Agreement): a contract with financial penalties. Your SLA should always be looser than your SLO, so you breach your internal target long before you owe anyone money.

The order matters. Teams that start with the SLA and work backwards end up measuring what lawyers wrote, not what users experience. Start with the SLI.

Choose SLIs that measure user pain

A good SLI moves when users are unhappy and stays flat when they are fine. CPU utilization fails that test. So does "server is pingable." The SLIs that work are ratios of good events over valid events, measured as close to the user as your instrumentation allows.

For a typical API-backed product, we usually start with three:

• Availability: proportion of requests that return a non-5xx response. Exclude 4xx: a user typo is not your outage.
• Latency: proportion of requests faster than a threshold. Not the p99 value itself, but the percentage of requests under, say, 400 ms. This framing composes cleanly into a budget.
• Correctness or freshness for pipelines: proportion of records processed within their deadline, or checksum-verified writes.

Two hard-won rules. First, measure at the load balancer or edge, not inside the application. A crashed process reports nothing, and an SLI computed from app metrics will happily show 100 percent while the service is down. Second, define validity carefully. Health check traffic, bot noise, and retries can flatter or slander your numbers.

In Prometheus, an availability SLI is a recording rule:

- record: sli:api_availability:ratio_rate5m
  expr: |
    sum(rate(http_requests_total{job="api", code!~"5.."}[5m]))
    /
    sum(rate(http_requests_total{job="api"}[5m]))

For latency, use the histogram bucket counts directly: requests under the threshold divided by all requests. This is cheaper and more honest than quantile estimation, since histogram_quantile interpolates and can mislead near bucket boundaries.

Error budgets: the contract that ends the war

An SLO of 99.9 percent over 30 days gives you a budget of 0.1 percent of requests, or about 43 minutes of full downtime. That budget is a shared account between product and engineering, and it settles the ship-versus-stabilize argument mechanically:

• Budget healthy: ship aggressively. Deploy daily. Run the risky migration. Velocity is the priority, and reliability work can wait.
• Budget exhausted: feature freeze. All engineering effort goes to reliability until the budget recovers. Not as punishment, but because the data says users are hurting.

This only works if leadership signs the policy before the first breach. An error budget policy that gets renegotiated mid-incident is theater. The policy should be one page: who owns the SLO, what happens at 50 percent consumed, what happens at 100 percent, and who can grant an exception (ideally one named executive, in writing).

The less obvious benefit runs the other way. If you consistently finish the month with 90 percent of your budget unspent, you are over-investing in reliability. Loosen the target, or better, spend the surplus: deploy more often, delete the flaky pre-production sign-off, run that failover test in production. Unspent error budget is unspent velocity.

Alert on burn rate, not on symptoms

The classic mistake is alerting on the SLI threshold itself: "page if availability drops below 99.9 percent in the last 5 minutes." That pages someone for a 5-minute blip that consumed 0.01 percent of the monthly budget, and teaches the on-call to ignore pages. Alert fatigue is how error budgets cause burnout instead of preventing it.

The fix is burn rate: how fast you are spending budget relative to the sustainable pace. A burn rate of 1 means you will spend exactly your budget by the end of the window. A burn rate of 14.4 against a 30-day window means you exhaust the entire monthly budget in about two days, which absolutely deserves a page.

The pattern that works in practice is multi-window, multi-burn-rate:

• Page: burn rate above 14.4 over 1 hour (consumes roughly 2 percent of monthly budget). Fast, severe, human needed now.
• Page: burn rate above 6 over 6 hours (roughly 5 percent of budget). Slower bleed, still urgent.
• Ticket: burn rate above 1 over 3 days. A slow leak that should be fixed this week, not tonight.

Each alert pairs a long window with a short control window, so the alert also resolves quickly once the problem stops:

- alert: ErrorBudgetFastBurn
  expr: |
    (1 - sli:api_availability:ratio_rate1h) > (14.4 * 0.001)
    and
    (1 - sli:api_availability:ratio_rate5m) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "API burning >2% of 30d error budget per hour"

Generating these by hand for every service gets tedious and error-prone. Tools like Sloth compile a compact SLO spec into the full set of Prometheus recording and alerting rules. OpenSLO gives you a vendor-neutral YAML spec if you want portability. Datadog and Grafana both ship native SLO features with burn rate alerts built in, and for most teams the managed version is the right call. The math is standard. Your differentiation is choosing good SLIs, not reimplementing the arithmetic.

This is also where observability stops being a dashboard hobby and starts paying rent: burn rate tells you that you are in trouble, and traces tell you why. If your telemetry cannot answer "which endpoint, which dependency, which deploy," start with our guide to logs, metrics, and traces before layering SLOs on top.

Failure modes we see in real teams

• Too many SLOs. Twelve objectives per service means none of them matter. Start with one or two per user-facing journey. You can always add more.
• SLOs on internal metrics. CPU, queue depth, and pod restarts are diagnostics, not objectives. If it cannot be explained to a user, it is not an SLO.
• The 100 percent SLO. Usually smuggled in by a well-meaning manager. It means every blip is a crisis and the error budget is zero forever. Refuse it.
• Calendar-window gaming. Monthly-reset budgets invite "it resets on the 1st, ship it" behavior. Rolling windows (30 days, sliding) remove the cliff.
• SLOs without a policy. A dashboard nobody acts on. If budget exhaustion does not change next sprint's content, you have monitoring, not SLOs.
• Ignoring the deploy pipeline. In our experience a large share of budget burn traces back to changes. If deploys are the leading cause, budget policy should gate deploy risk first: smaller batches, canaries, automatic rollback. Our piece on zero-downtime deployments covers the mechanics.

A 30-day rollout, step by step

You do not need an SRE department to start. You need one service, one owner, and a month:

1. Week 1: pick one user journey. The checkout call, the login flow, the search endpoint. Instrument availability and latency at the edge, and backfill from load balancer logs if you have them.
2. Week 1: measure before you target. Run the SLI for a week with no objective attached. Guessing a target before seeing data produces fiction.
3. Week 2: set the SLO just below current performance. If you measured 99.93 percent, set 99.9. The first SLO should be achievable, because the habit matters more than the number.
4. Week 2: write the one-page error budget policy and get it signed by engineering and product leadership. Named owner, named escalation, named exception authority.
5. Week 3: wire up burn rate alerts with the multi-window pattern. Delete the threshold alerts they replace. Every page must be actionable, and your paging flow should hand off cleanly to your incident response playbook.
6. Week 4: hold the first SLO review. Thirty minutes, monthly, standing agenda: budget consumed, biggest burn events, one decision (tighten, loosen, or hold). This meeting is where SLOs become culture instead of config.

After the first quarter, expand to a second service and revisit targets. Tighten only when users complain at the current level, loosen when you consistently leave budget on the table.

The burnout connection is not a metaphor

The subtitle of this post is a claim worth defending. On-call burnout comes from two places: pages that do not matter, and reliability arguments that never end. Burn rate alerting attacks the first directly, because every page now maps to real budget loss and real user pain. The error budget policy attacks the second, because "should we slow down" becomes a lookup, not a negotiation.

There is a third effect that shows up after a few months. Engineers stop being the people who say no. When the budget is healthy, engineering says yes to the risky launch with data behind the yes. When it is spent, the freeze is the policy's decision, not a personality conflict. That depersonalization is worth more than any dashboard.

How Innovation T can help

Innovation T designs and runs this stack for clients across Europe and North Africa: SLI instrumentation, Prometheus and Grafana or Datadog SLO tooling, burn rate alerting, and the error budget policies that make it stick. We build the cloud and DevOps foundations first, then the reliability practice on top, so you get SLOs your team actually uses instead of dashboards nobody opens.

If your on-call rotation is tired and your uptime target is still a slide from 2023, talk to us. We will start with one service and one SLO, and you will feel the difference in a quarter.