Skip to content

SRE-Error-budget

SRE 1. Is error budget realistic. 2. What are the single source of burning off Error budget 3. Any low hanging fruits that can increase availability

Identify Hazards and calculate potential risks due the hazard. Eg: Loadbalance over load. Categorize risks based on classes of risks Prioritize risks based on their potential impact

Risk = Probability * impact

Downtime is total of

  • Time to detection
  • Time to resolution
  • Percentage of users impacted

Documenting SLO

SLO defines threshold/line which defines reliable/unreliable service Must include * Why the threshold is where it is * Why the SLI are appropriate for measuring the SLO * Identify monitoring data deliberately excluded from the SLIs.

Track SLO version and evaluate it periodically

Error Budget Policy

An error budget policy describes how your business decides to trade off reliability work against other feature work when an SLO indicates a service is not reliable enough. The purpose of such a policy is to guide your organization into meaningful and appropriate action when the reliability of your service is threatened.

Key considerations for an effective error budget policy:

1. Trigger conditions — Be clear about when the policy takes effect. This could be when you've exhausted your trailing four-week budget or when your error budget is at risk (e.g., 50% of the monthly budget burnt in one week). It is reasonable to have multiple trigger points with different actions depending on severity.

2. Response actions — Detail how dev and SRE teams will prioritize reliability features. Tie the response to the severity of the miss. For example, if the budget is threatened but not exhausted, commit one developer to fixing all high-priority post-mortem items. If sustained exhaustion spans months, the entire dev and SRE teams work solely on reliability until the budget is sufficiently replenished.

3. Consequences — To give the policy real teeth, there must be meaningful consequences for consistently failing to meet an SLO over longer time horizons. A team focused solely on features at the expense of reliability is not serving its users well. At Google, SRE teams can return the pager to the dev team if reliability is not being effectively prioritized. Google's "Code Yellow" protocol is another example: any engineer can request it, and with executive agreement, a dedicated team focuses entirely on mitigation and resolution.

4. Consistent application — Carving out special cases for one team opens the door to others demanding the same and undermines the SRE team's ability to maintain reliable service. Where business priorities dictate exceptions (e.g., a contractual obligation), a limited number of "silver bullets" — once or twice per year — can allow feature work to continue despite an exhausted SLO. Each silver bullet must be followed by a postmortem focused on why it was necessary.

5. Clear decision maker — Disagreements between teams are inevitable. The policy must name who makes the final call when escalations occur. Executives should back the policy and enforce its consequences.

6. Agreed and signed off — The most important part of the policy is not its contents, but that all parties — product managers, developers, SREs, and executives — know about it and have agreed to abide by it. A policy no one agreed to is difficult to enforce.

Policy Thresholds

  • Threshold 1 Automated alerts notify SRE of an At-risk SLO
  • Threshold 2. SRE Conclude they need help to defend SLO and escalate to Devs
  • Threshold 3. The 30day error budget is exhausted and the root cause has not been found; Feature releases blocked, dev team dedicated more resources.
  • Threshold 4. The 90 day error budget is exhausted and the root cause has not been found. SRE ** escalates to executive leadership ** to obtain more engineering time for reliability work.