Lesson 9 of 9 · about 15 minutes
Security, Reliability and Why Outages Happen
You'll be able to
- Describe the layers of physical security at a typical data center
- Cite Uptime Institute figures on outage frequency, causes and costs
- Explain why human error and software, not just equipment, cause many outages
- Read a provider's post-incident report and pull out cause, duration and scope
Most data centers look deliberately boring from the road: a long, windowless building, no sign or a vague one, a tall fence set back from a berm. That is the first layer of security. If a building does not announce what it holds, fewer people get curious.
Layers of physical security
Security people call it defense in depth. An intruder, or a careless contractor, should have to pass several separate barriers before touching a server.
- Perimeter. Fences, often anti-climb; crash-rated bollards or berms to stop vehicles; a single guarded gate; cameras covering the fence line.
- Building entry. A staffed security desk, government ID checks, pre-registered visitors only, escort rules.
- Mantraps. Two interlocking doors so only one verified person passes at a time, stopping tailgating.
- Badge plus biometrics. Card readers combined with fingerprint, palm vein or iris scans at the data hall doors.
- Cages and rack locks. In colocation, each tenant's gear sits in a locked cage or rack.
- Logs and cameras. Every door event is recorded; video is kept for a set period. Auditors for standards such as SOC 2 and ISO/IEC 27001 check these records.
- Media destruction. Failed drives are shredded or degaussed on site so data never leaves intact.
Physical security also covers the things a building depends on: fuel deliveries, water supply, fiber routes and the substation. A cut fiber bundle at a construction site down the road can do more damage than a person at the front gate, which is why sites insist on fiber entering from different directions.
What the outage numbers say
The Uptime Institute surveys operators every year. The 2025 global survey and the 2026 outage analysis give the clearest picture available, with the usual caution that survey answers are self-reported.
| Measure | Figure | Source |
|---|---|---|
| Operators with an impactful outage in the past three years | 50% in 2025, down from 53% in 2024 and 74% in 2020 | Uptime 2025 survey, 2026 outage analysis |
| Most recent impactful outage rated serious or severe | About 1 in 10 | Uptime 2025 survey |
| Power as cause of most recent impactful outage | 45% in 2025, down from 54% in 2024 | Uptime 2026 outage analysis |
| Most recent major outage cost over $100,000 | 57% | Uptime 2026 outage analysis |
| Most recent impactful outage cost over $1 million | About 1 in 5 | Uptime 2026 outage analysis |
| Human error at least a minor contributor | 92% | Uptime 2026 outage analysis |
| Outage preventable with better management, processes or configuration | 87% | Uptime 2026 outage analysis |
Outages are becoming less frequent, but the expensive ones are not shrinking, because more of the economy depends on fewer, larger facilities. And the root of most outages is not exotic. A breaker fails and a transfer does not go as planned. Someone skips a step in a procedure. Uptime's 2025 analysis said 85 percent of human-error-related major outages involved staff not following procedures, or procedures that were wrong.
Why redundant systems still fail
Lesson 5 described N+1, 2N and Tier IV. Those designs protect against a single equipment failure. They do less against:
- Software and configuration. Automated systems can push a bad change everywhere at once, defeating physical redundancy.
- Common-mode failures. One cause hits all the redundant parts together: a firmware bug in every UPS, a fuel contamination problem in every generator, a heat wave beyond the cooling design.
- Maintenance windows. When one side of a 2N system is out for service, the site runs on the other with no spare.
- Human steps. Switching sequences done in the wrong order.
Worked example: reading a real post-incident report
AWS published a summary of its October 19 to 20, 2025 disruption in the Northern Virginia (us-east-1) Region. Pull out the basics:
- Start: 11:48 p.m. PDT, October 19
- Trigger: A latent race condition in DynamoDB's automated DNS management. An older plan overwrote a newer one, and a cleanup process then deleted it, leaving the regional endpoint with no IP addresses.
- Trigger resolved: DNS records restored by 2:25 a.m. PDT, October 20, with cached entries cleared by 2:40 a.m., about 2 hours 52 minutes after the start
- Full recovery: 2:20 p.m. PDT, October 20, as knock-on effects in EC2, load balancers and other services cleared
- Total duration: 11:48 p.m. to 2:20 p.m. is 14 hours 32 minutes
Notice what is missing: no fire, no power failure, no flood. The buildings were fine. A software fault in one service cascaded through others that depended on it. For anyone trying to make sense of it, the useful point is that cloud outages are more often logic failures than physical ones, and that heavy concentration in one region spreads the damage widely.
Grid events and the data center
Reliability runs in both directions. NERC's review of the July 10, 2024 event in the Eastern Interconnection found that about 1,500 MW of data center load switched to backup power after repeated voltage dips, and that about 1,260 MW stayed off the grid for hours. The data centers stayed up. The grid had to manage a sudden drop in demand it had not planned for. As campuses grow toward gigawatts, grid operators treat their behavior during disturbances as a reliability issue in its own right.
Common mix-ups
- Claim: Big outages are usually caused by hackers. What the evidence shows: Uptime's data points to power, network, software and process failures as the most common causes. Cyber incidents are rising, but most outages are self-inflicted.
- Claim: More redundancy always means fewer outages. What the evidence shows: Redundancy adds complexity, and complexity creates new ways to fail. Most operators say better procedures would have prevented their last outage.
Reporting angle
- When a cloud service fails, go to the provider's status page and later its post-incident summary. Quote times and causes from the provider, and say when details are still unconfirmed.
- Ask local agencies, schools and hospitals which cloud regions their critical systems use and whether they have a fallback.
- For a local facility, ask the fire marshal and emergency management office about battery storage, fuel storage and response plans. Lithium-ion battery rooms are covered by fire codes such as NFPA 855.
- Ask the utility whether large data center loads must meet ride-through requirements, and whether any local load loss events have been reported.
Research angle
- Uptime Annual Outage Analysis 2026: Causes, costs and frequency of outages.
- Uptime outage analysis 2025 release: Announcement with the headline findings.
- AWS US-EAST-1 disruption summary: A provider's own post-event account.
- NERC large load loss review: Incident review of data centers dropping off the grid at once.
- Uptime Tier Classification: What each tier level claims and how sites get certified.
- Uptime Global Survey 2025 report: Rack density, PUE history and staffing findings.
Pictures, Numbers and Infographics
U.S. data centers tripled their electricity use in ten years, and the federal estimate for 2028 is roughly double to triple the 2023 level. The buildings got more efficient for years, then stalled, as the PUE chart shows.
By the numbers
- 58 to 176 terawatt-hours: U.S. data center electricity use in 2014 and in 2023. The 2028 projection is 325 to 580 terawatt-hours. U.S. Department of Energy and Lawrence Berkeley National Laboratory, 2024
- 4.4%: data centers' share of all U.S. electricity in 2023, expected to reach about 6.7% to 12% by 2028. U.S. Department of Energy and Lawrence Berkeley National Laboratory, 2024
- 5,427: data centers in the United States, more than 10 times as many as any other country. Stanford AI Index, 2026
- 1,136: hyperscale data centers worldwide at the end of 2024, double the count of five years earlier. The U.S. held 54% of their capacity. Synergy Research Group (industry analyst), 2025
- 1.54: weighted average PUE reported by operators in 2025, the sixth straight year the figure has barely moved. It was 2.5 in 2007. Uptime Institute (industry survey), 2025
- Almost 9 kW: average typical power per rack in the 2025 survey, up from 8.3 kW in 2024. Uptime Institute (industry survey), 2025
Check yourself
1. In Uptime Institute's 2026 outage analysis, what was the leading cause of impactful outages in 2025?
Power was the leading cause at 45 percent of respondents' most recent impactful outages, down from 54 percent the year before.
2. What share of respondents said their most recent major outage cost more than $100,000?
Uptime reported 57 percent over $100,000 and about one in five over $1 million.
3. What caused the AWS us-east-1 disruption of October 19 to 20, 2025, according to AWS's own summary?
AWS said two automated processes interacted badly, deleting the DNS record for the regional DynamoDB endpoint, and the failure cascaded to other services.
4. What is a 'mantrap' at a data center entrance?
Mantraps stop tailgating by letting one door open only after the other closes and the person is verified.
5. Uptime says 87 percent of respondents with an impactful outage believed it could have been prevented by what?
Most outages involve human and process factors, so better procedures and configuration are seen as the main fix.
Terms in this lesson
- Mantrap
- A secure vestibule with interlocking doors that admits one verified person at a time.
- Defense in depth
- Layered security, where an intruder must pass several separate barriers to reach the equipment.
- Impactful outage
- Uptime Institute's term for an outage with significant effects on services, customers or finances.
- Post-incident report
- A provider's written account of an outage, covering timeline, cause and fixes.
- Cascading failure
- A failure in one system that triggers failures in others that depend on it.
Sources
- Uptime Institute: Annual Outage Analysis 2026 (PDF) checked 2026-10
- Uptime Institute: Annual Outage Analysis 2025 announcement (Business Wire) checked 2026-10
- Uptime Institute Global Data Center Survey 2025 (report PDF) checked 2026-10
- AWS: Summary of the Amazon DynamoDB service disruption in the Northern Virginia (US-EAST-1) Region checked 2026-10
- NERC: Incident Review, large load loss (July 10, 2024) checked 2026-10
- Uptime Institute: Tier Classification System checked 2026-10