Darragh Kennedy (@darraghke) header
Darragh Kennedy (@darraghke) profile picture

Director of Engineering @Zendesk - views are my own

Melbourne
Joined April 2009
Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)Photo by Darragh Kennedy (@darraghke)
Darragh Kennedy retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations. The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services. First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments. The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts. Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own. Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States. When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally. Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway. Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change. After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region). The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
164
368
94
2,778
480,108
Darragh Kennedy retweeted
Banger new guide from Anthropic. It discusses the core primitives of always-on agents. It covers the importance of an event inbox, using timers for proactive capabilities, ensuring continuity with memory, and more. Read more: claude.com/resources/article…
91
55
9
454
36,145
Darragh Kennedy retweeted
Tldr: - Very bad incident (most sensitive client data exposed) - Cursor blames Fireworks (leaked an important Cursor key following some mutual work they had) - Great report from Cursor overall, moved off Fireworks inference due to this
45
53
6
1,682
308,697
Darragh Kennedy retweeted
Faster customer-service work creates a commercial choice: where will the released capacity go? McKinsey's diagram shows several possible uses: maintain service with fewer resources, serve more customers, move into customer success, or combine service with quality and knowledge work. These are different business choices. Serving more accounts may support expansion. Giving people time to resolve recurring customer problems may improve the service proposition. Reducing staffing may release cost while limiting either option. The chart is illustrative. Its capacity shares are not measured productivity gains or a staffing recommendation. Otherwise, the organisation can report time saved while the intended customer benefit remains unstaffed. Source: McKinsey & Company, “AI is changing work. Now it has to change the organization” @RLDI_Lamy @sulefati7 @ILoveBooks786 @Corix_JC @FrRonconii @Nicochan33 @faryus88 @ramonvidall @ozsilverfox @jameslhbartlett @dinisguarda @PatrickCFOD @Eli_Krumova @BCAgroup @ricardo_ik_ahau @timo_vi @violetainsights @DioOmega @66Iot
2
9
15
291
Darragh Kennedy retweeted
What four tensions did we uncover when studying the use of AI in @Google engineering? Here's a great article on these systemic tensions, how to mitigate, and why good team dynamics paired with anti-debt and workflow focus is a recipe for success. ieeexplore.ieee.org/document…
1
3
34
2,833
Today, we’re shipping updates to Amazon Bedrock AgentCore Runtime that help customers meet the demands of agents taking on longer, bigger, and more autonomous work. Agent sessions can now start up to 15x faster, scale to 4x more compute, and run up to 68% more efficiently, without giving up isolation. aws.amazon.com/blogs/machine…
4
14
2
81
11,856
Darragh Kennedy retweeted
deploying persistent agents like Openclaw or Dots pose a major challenge for enterprise IT agents are more like contractors than traditional saas software, able to take proactive actions in the background without human intervention they are capable of fulfilling entire job functions but they can also take misaligned actions and write arbitrary code with unintended consequences today, we are sharing some of the best practices we use to manage fleets of claws at @OpenAI developers.openai.com/blog/s…
48
82
24
965
180,732
Darragh Kennedy retweeted
MCP or CLI? Instead of debating which is better for agents, we're shipping both in one flow. `sentry mcp` launches the MCP server in stdio mode and shares the credentials you already set up with the CLI, so one device-code login covers both. The post covers why we think you need both: → Shared auth gives us control of the login flow, though consolidated MCP auth is still the better long-term answer → The CLI has a filesystem, so agents can grep and paginate raw data like session replay events, which MCP can’t do efficiently → and more that you should read in the full post We’re also moving onboarding toward a CLI-based approach, and we’d love your feedback if you try it 🤝
2
25
2,539
Darragh Kennedy retweeted
Nine engineers. 1,211 pull requests in a day. Go behind the scenes of building Copilot Cowork for six lessons on engineering with agents, including how the team moved its runtime from Python to Rust while the product was live. commandline.microsoft.com/ho…
17
36
6
284
23,626
Darragh Kennedy retweeted
Long-horizon agents can automate tasks that take days or weeks, but much can go wrong. Kiran Prakash explains how durable execution allows agents to recover. martinfowler.com/articles/du…
16
40
5
274
25,484
Darragh Kennedy retweeted
When Anthropic release a new model it's always worth taking a look at their updated prompting guide - here's an entertaining tip to make Fable 5.2 less likely to be cautious in how it improves existing code: tell it "This codebase is yours" platform.claude.com/docs/en/…
439
322
151
4,732
370,691
“Wiz counted at least nine AI developer platforms that disclosed an incident in October alone, from Fireworks AI on the 6th to Northflank on the 30th. Fireworks, OpenCode, and Warp have all reported indications of an "agent swarm" attack. The motive is that a stolen inference key can be spent immediately: Unit 42 has seen bills reach nearly $1 million before detection.”
Everyone is getting hacked and everyone is going to get hacked It’s only accelerating and going to accelerate more and more with agents Please, please go talk to your team about this If they aren’t taking it extremely seriously, you need to drive change. You will be hacked.
14
6
2
43
10,370
Darragh Kennedy retweeted
🧠 Most AI bug-catching tools review your code the way a human reviewer would, only faster. What if the AI wrote a test that proved the bug instead? That’s Catching Just-in-Time Tests (JiTTests) — a novel approach where LLMs generate executable tests designed to fail, surfacing behavioral changes in a code change before it lands. After a year of research, it’s now part of Meta’s infrastructure. ✅ 589 engineers have engaged with it, reporting a 60% true-positive rate 📈 In the first half of 2026, it helped catch 74 privacy SEVs at the diff stage, lowering overall privacy SEV occurrence by 20% 🔗 Read how it works: engineering.fb.com/2026/10/2… 📄 Read our FSE 2026 paper, “Just-in-Time Catching Test Generation at Meta”: arxiv.org/pdf/2601.22832 #AI #SoftwareEngineering #SoftwareTesting #DeveloperTools #Engineering
35
79
15
771
44,849
Darragh Kennedy retweeted
Google、Amazonなどの企業がオンコール体制をどう構築しているかを解説した記事。従来の開発と運用の分離から、開発者が自らコードの運用やオンコールを担う「You build it, you run it」モデルへの移行の歴史と背景。 increment.com/on-call/who-ow…
14
89
3,619