Kelly J. Rose (kjrose)

Tomorrow is the last Grey Cup ever played on a 110-yard field with the goalposts on the goal line. The math guy in me is grieving the 55-yard line. If the Elks win it, I want one last doink off the upright for old times' sake.

Official cause of the FAA ground stop: legacy routers got "overloaded with data" while new gear was being patched in. So nobody hacked the router. It just ate too much Thanksgiving dinner. Check what's still humming in your own server closet before your next upgrade.

FAA ground stop lifted after 4+ hours. Official cause: "telecommunications outage." Reported cause: one network update rebooted the whole thing. Nearly 5,000 flights cancelled. Have you tried turning it off and on again? Yes. It took four hours.

Lol. This did not age well... At all.

Sources tell CBS and the WSJ an update to the FAA comms network run by L3Harris triggered a reboot, then a crash. Unconfirmed. But if true, someone pushed a change to every air traffic centre in the country at once, on a Friday. Canary deployments exist for a reason.

All 22 US air traffic centres went dark at once over a "telecommunications outage." 22 centres, apparently 1 phone bill. If your business runs on one ISP, one carrier and one prayer, today's a free tabletop exercise.

The UK's NCSC is warning about scammers posing as Microsoft or your IT help desk off the back of the M365 outage. Rule of thumb: Microsoft is not going to phone you, and your IT people will never need your password or MFA code. If someone calls about "fixing your Outlook," hang up and call us back on a number you already have.

Someone at the FAA Today.

Microsoft 365 falls over for 30 hours. Next day, every US air traffic centre loses comms at once. I'm sure it's a coincidence. I'm also sure I'll be checking my backup generator tonight. Boring infrastructure is boring right up until it isn't.

Microsoft 365, hour 21: North America is finally improving. Microsoft says delayed mail is being kept for 48 hours, so yesterday's emails are still coming, just very late, like a Friday-afternoon reply to a Tuesday question. Don't re-send everything. Let the backlog drain.

Microsoft 365 outage, hour 12 and change: Microsoft now reports "partial recovery" in some regions, then admits it isn't sticking. The servers have discovered the rally where you score one goal and then take three penalties. If it matters tonight, call it in.

Hour 11 of the Microsoft 365 outage: Microsoft says its latest capacity fix "is not taking as expected," and it's finding "alternate workstreams" for "sustained traffic management imbalances." So it's the Whitemud at 5 PM, but for email. Next update by 9:30 PM MST.

Hour 10 of the Microsoft 365 outage: Microsoft says regions are "coming online" and it's adding capacity to handle the traffic. It's like watching a strategy game map get unfogged one tile at a time. It's not fixed everywhere yet. Next update by 9 PM MST.

Hour 9.5 of the Microsoft 365 outage: Microsoft reports "positive results" from restarting the affected servers. A billion-dollar cloud, and the breakthrough was turning it off and on again. It's not fixed yet; they're checking stability before expanding. Next update by 9 PM MST.

Hour 9 of the Microsoft 365 outage. Microsoft says the process that sets up secure connections got overwhelmed and quit. Exchange shook too many hands and faked a sore wrist. A fix is in testing and a network change is being rolled back. Next update by 8:30 PM MST.

M365 update, 7 PM MT: the trouble is anything tied to Exchange and email. Teams chat and meetings mostly work, and SharePoint files open. No fix time yet. Probably someone vibe coded it and skipped the PANDAROSE bug review. 😉

Microsoft 365 setback: servers brought back online are falling over again once traffic hits. As of 5:59 PM MT, Microsoft is limiting traffic and rebuilding some servers on an older config. Next update around 7 PM MT. Outlook tonight is a lawn chair in a windstorm.

Microsoft 365 update: some servers are back and holding steady. Recovery is gradual, so Outlook may come back one sad little ping at a time. No all-clear yet. If your inbox suddenly gets 400 emails at once, that's the good news.

Microsoft 365 update, 4 PM MT: Microsoft says its fixes so far haven't given the relief it hoped for. It's now isolating chunks of its infrastructure, restarting them and easing traffic back in. Outlook, Teams and SharePoint may stay flaky into the evening. Grass remains available for touching.