High availability: why 99.9% uptime is not enough

Why "uptime" is the metric that lies the most

"We have 99.9% availability." It's a sentence that sounds like an achievement, but hides an uncomfortable truth: 99.9% means a service can be unavailable for 8 hours and 45 minutes per year and still meet its promise. For an online store that bills €10,000 per hour, that number translates into up to €87,500 of revenue at risk annually — without anyone having formally failed at anything.

Availability is not an abstract percentage: it's time, money, and trust. This article breaks down what the "nines" really mean, introduces the concept of error budget used by Google's site reliability engineering teams, and shows — concretely, with the architecture of a real cluster — how to build systems that stay standing when things go wrong.

What the "nines" really mean

The industry measures availability in "nines". Each additional nine divides the allowable downtime by (almost) ten. The difference between 99.9% and 99.99% is not "a little better" — it's going from 8h 45m to 52 minutes of downtime per year. The table and chart below show the maximum allowable interruption duration for each tier.

SLA level	Downtime / day	Downtime / month	Downtime / year
99% (2 nines)	14 min 24 s	~7 h 18 min	3 days 15 h 36 min
99.9% (3 nines)	1 min 26 s	~43 min 48 s	8 h 45 min 36 s
99.95%	43 s	~21 min 54 s	4 h 22 min 48 s
99.99% (4 nines)	8.6 s	~4 min 22 s	52 min 33 s
99.999% (5 nines)	0.86 s	~26 s	5 min 15 s

Notice the detail: the jump from 99.9% to 99.999% reduces allowable downtime a hundredfold. That's why each additional "nine" costs exponentially more in infrastructure, processes, and team.

The cost of downtime

The financial impact varies drastically depending on the business. Using an illustrative example of a service that generates €10,000/hour in revenue, the annual potential cost of the downtime allowed by each SLA is as follows:

SLA	Downtime/year	Potential cost/year (at €10k/h)
99%	3 days 15 h 36 m	~€876,000
99.9%	8 h 45 min	~€87,600
99.99%	52 min	~€8,760
99.999%	5 min	~€876

But the cost isn't only direct. There are indirect costs that are hard to quantify: loss of trust, customer churn, a saturated support team, unmet contractual SLAs (with penalties), and the reputational effect that lasts long beyond the interruption itself.

SLI, SLO, and SLA — and the error budget

To manage availability rigorously, Site Reliability Engineering teams use three distinct concepts, popularized by Google in the book Site Reliability Engineering:

• SLI (Service Level Indicator) — the objective measurement: for example, "successful HTTP requests / total requests".
• SLO (Service Level Objective) — the internal target: "99.9% of requests successful, over a 28-day window".
• SLA (Service Level Agreement) — the external contract with the customer, typically less ambitious than the SLO, leaving a safety margin.

The concept that ties it all together is the error budget: the difference between 100% and the SLO. If the SLO is 99.9%, the monthly error budget is ~44 minutes. While there is budget, the team can ship new features (take on risk); when the budget runs out, the focus shifts to stabilization. It's an elegant mechanism that turns reliability into a product decision, not a technical dogma.

How high availability is built, in practice

High availability doesn't come from "one really good server". It comes from removing single points of failure at every layer — and automating recovery.

1. Redundancy at every layer

Every component should have replicas: parallel load balancers, multiple application replicas, databases in primary-replica (or multi-primary), and replicated storage. If one element dies, another takes over without manual intervention.

2. Orchestration with Kubernetes

Systems like Kubernetes (k3s, EKS, GKE, GKE) offer, out of the box, the resilience primitives:

• Auto-healing — a failing pod is automatically recreated on another node;
• Rolling updates — new versions roll out gradually, without outages;
• Health checks (liveness/readiness) — traffic only goes to healthy instances;
• Replication and scaling — the number of replicas adjusts to the load.

3. Resilient storage

Data is, almost always, the most valuable asset. Distributed storage like Longhorn, Ceph, or Rook replicates each volume across multiple nodes; if a disk or a machine fails, the data remains accessible. Regular snapshots provide rollback and protection against attacks (including ransomware).

4. Multi-zone / multi-region

For true resilience, the service exists in more than one availability zone. If a data center goes down — due to power failure, fire, or cut fiber — traffic is rerouted to another. This is expensive, so it's reserved for genuinely critical workloads.

5. Observability and proactive response

You can't improve what you don't measure. Stacks like Prometheus + Grafana + Alertmanager, or services like Uptime Kuma and Datadog, monitor SLIs in real time and fire alerts before the user notices. Define runbooks and do chaos engineering (e.g., shut down a node on purpose, during a quiet period, to validate recovery).

On your own cluster: the Raspberry Pi as a lab

You don't need a public cloud to practice these principles. A three-Raspberry Pi cluster running k3s, with Longhorn (volumes replicated across nodes) and Traefik + cert-manager, already implements most of what was described: auto-healing, rolling deploys, data redundancy, and automatic TLS. It's an excellent environment to internalize, at low cost, the patterns that later scale up to professional infrastructure.

Conclusion

99.9% isn't "almost perfect" — it's a compromise that tolerates almost 9 hours of downtime per year. The right question isn't "what's our uptime?", but rather: "what's the SLO our business requires, and how is it being measured and defended?". High availability is built layer by layer, with redundancy, automation, and observability — and it always starts with looking honestly at the numbers.

References

• Google — Site Reliability Engineering (book, free): chapter Service Level Objectives and the concept of error budget — sre.google/sre-book
• AWS — Well-Architected Framework, Reliability pillar — docs.aws.amazon.com/wellarchitected
• Microsoft Azure — SLA summary by service — azure.microsoft.com/support/legal/sla
• B. Beyer et al., The Site Reliability Workbook, O'Reilly / Google (practical application of SLOs and error budgets).