Snow Pond Tech Group

Lots of hot takes on the Microsoft 365 outage. Here's an engineering one. To be clear: this is speculation. Microsoft hasn't confirmed a root cause. It's my read of their own updates, from 35 years of building and breaking systems. What Microsoft has actually said: • Traffic isn't up. Capacity is down. Servers are failing health checks and dropping out. • A process that establishes secure connections gets overwhelmed, and servers stop processing requests. • Servers they bring back fall over again as soon as traffic hits them. • They're reverting a network config change that lines up with when it started. But changes aren't syncing, and some machines are stuck in a bad state. • No indication of an attack. My hypothesis: a metastable failure. Whatever triggered it may be long gone. The outage is now keeping itself alive. 1. Reconnect storm. Every Outlook, Teams and mobile client on the planet is hammering for a fresh secure connection. Full TLS handshakes are CPU-heavy, and a freshly rebooted server has a cold session cache. A server comes up, eats a wall of handshakes, CPU pegs, it fails its health check and gets pulled, and its load lands on the next one. Dominoes. 2. The fix shares fate with the failure. If the rollback travels over the same secure channel that's melting, the cure can't reach the patient. That's exactly what "changes aren't syncing" sounds like to me. That's why they're throttling traffic types and rebuilding servers from an older config instead of flipping a switch. You don't get out of this by finding the bug. You get out by shedding load, restoring capacity in isolated cells, and letting traffic back in slowly. Where AI fits (also speculation): not "an AI wrote the bad line". It's velocity. Microsoft flagged several recent config changes as suspects and took hours to find one that correlated. When the volume of change outpaces the humans who understand each change, finding the culprit gets slow and rollbacks get messy. Automate the shipping faster than you automate the understanding, and this is the bill. The takeaway for every business: your email, chat, files, identity and security tooling now share one failure domain. Today proved it. Have a way to talk to your team that doesn't live in the same tenant. Happy to be proven wrong by the post-incident review. And I hope this post ages terribly.