@︎№#ident𝚍 retweeted
Replying to @merill @theartofro
"EVERY NON-STANDARD CHANGE NEEDS CAB"=Classic blunder of failing to correctly read ITILv3 [what y'all gonna pretend yourself running v4 or v5? Don't try to pull wool over my eyes.] (2011) Service Transition core publication section 4.2.5.5 figure 4.5. Drives ~80% of F500 IT wonk.
"due to a separate issue"
THERE ARE FOUR LIGHTS, NOT THREE. WHICH ISSUE CAUSED THE "BULKHEADS" TO FAIL, MSFT.
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.
First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.
Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.
Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.
Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.
When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.
Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway.
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.
After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).
The final Post-Incident Review (PIR) will provide more details later this week.
31
@︎№#ident𝚍 retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.
First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.
Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.
Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.
Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.
When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.
Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway.
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.
After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).
The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
@︎№#ident𝚍 retweeted
Today an LLM hallucination resulted in 72 eggs being delivered to my house. 🥚🥚🥚🥚
@︎№#ident𝚍 retweeted
Alright now qwen 4 is here!
📢 Meet Qwen4-Max, your new AI teammate for work, creativity, and discovery!
Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window.
Highlights:
- Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation.
- Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables.
- Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery.
- $1.5/M tokens for input, $4.5/M tokens for output, and $0.075/M tokens for implicit caching — all lower than Qwen3.8-Max, with implicit caching down by 70%.
The API is live on QwenCloud. You can also try Qwen4-Max on Qwen Studio, QwenWork and Qoder.
Start building today! 🚀
- Blog: qwen.ai/blog?id=qwen4-max
- API: qwencloud.com/models/qwen4-m…
- Token Plan: qwencloud.com/pricing/token-…
More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀
@︎№#ident𝚍 retweeted
Elon through DOGE cut Ebola prevention and clinics that provided care to children with HIV resulting in the death of an estimated 500,000 children in ONE YEAR.
#MuskDocumentary
Six months after an Ebola outbreak was declared in Congo, the disease remains out of control and efforts to contain it are faltering. abcnews.link/DMdsSB4
@︎№#ident𝚍 retweeted
Grimes and Neuralink executive Zilis were in the same hospital just a hallway down from each other
#MuskDocumentary
@︎№#ident𝚍 retweeted
This Musk documentary is wild
Shivon Zilis (an executive at Neuralink) and Grimes were in the same hospital just a hallway down from each other.
Zillis was having Elon's twins while Grimes was there with her and Musk's surrogate. Grimes was unaware.
#Muskdocumentary
one idiotic web design is the horizontal scroll for code.
it was made popular by stackoverflow.
you should just display wrap instead.
this harm came from the unix and c idiots, line-based computing of 80 chars per line, which is from punched cards of 60s.
@︎№#ident𝚍 retweeted
Based on my many years covering this stuff I put the odds of @JensenHuang coming back at less than zero. The real story: Why are these tech/Wall Street dudes floating this nonsense? My belief is that it’s desperation. If they really believed in the AI trade they would simply say it’s time to move on from Jensen who was great etc. but it did some dopey things; the fundamentals of the company and the trade are fine and regulatory risk is already factored into the stocks. But they’re not doing that; quite the opposite. They’re panicking by floating this fantasy. Imagine for a moment Jensen coming back and testifying before the rabid anti AI Dems in Congress as the CEO of @nvidia who wanted to sell China his best chips and that’s before having to explain under oath the other stuff. Imagine him sitting before Bessent who just today said he was a great innovator but lacking in what it takes to run such an important enterprise. So just stop it; actually don’t. Please make Jensen coming back a reality for the sake of reporters like yours truly who will have an even bigger field day
@︎№#ident𝚍 retweeted
Polygenic evolution towards a morning chronotype in modern humans (preprint). 'We propose that the trend towards morningness is stronger in present-day modern humans living above 20 degrees north than in Neanderthals' biorxiv.org/content/10.64898…
@︎№#ident𝚍 retweeted
In other words the US government belatedly realized this could kill the stock market and are in full damage control mode.
BREAKING: Jensen Huang considering comeback as Nvidia's CEO, days after resigning, with Silicon Valley allies assembling a "war room" to lobby the board for his reinstatement, per FT.
The main reason for the reinstatement discussion is US intelligence concluding it is not established that Alice Huang was acting as a spy for China.
"No conclusions reached by the Department of Justice, or by me, or by Director Patel around who she is or is not," Attorney General Todd Blanche said.
Officials at the DNI have described the characterization of her as a Chinese spy as over-torqued, and the CIA is not convinced the intelligence collected proves she was knowingly passing information to China.
Scott Bessent and Marco Rubio, who were both part of the effort that led to the resignation, called on Sunday for a "dramatic shake-up" in leadership at the company.
@︎№#ident𝚍 retweeted
So fun history tidbit: “Stop quoting laws to MEN WITH SWORDS” is the Roman general Pompey's line, from when a city cited its legal rights to him.
Pompey was later stabbed and beheaded in Egypt so the swords didn’t exactly work out great.
@︎№#ident𝚍 retweeted
Quick heads up on an updated passkey registration experience.
TLDR Microsoft is making it easier for users to succesfully register any passkey to avoid hitting a brick wall.
I belive this also ends up guiding users to register synced passkeys (the webauthn flow) outside of Authenticator.
so darn bloody true YOU DON'T EVEN KNOW
The most hilarious CAB I ever sat on had a reviewer who was super pedantic about grammar and typos.
Every meeting he'd show up with paper printouts of the change doc, red scribbles all over.
One time he got so mad at the number of typos that he threw his notes at the poor team whose change he was reviewing and stormed out of the room.
It was glorious 😂
Another time I was working with a large customer on a fairly big Entra change to their tenant.
They asked for the roll-back plan. The Entra team went, 'we don't do roll-backs only roll-forwards.'
And that's when the fight started... 😛
1
79
@︎№#ident𝚍 retweeted
The most hilarious CAB I ever sat on had a reviewer who was super pedantic about grammar and typos.
Every meeting he'd show up with paper printouts of the change doc, red scribbles all over.
One time he got so mad at the number of typos that he threw his notes at the poor team whose change he was reviewing and stormed out of the room.
It was glorious 😂
Another time I was working with a large customer on a fairly big Entra change to their tenant.
They asked for the roll-back plan. The Entra team went, 'we don't do roll-backs only roll-forwards.'
And that's when the fight started... 😛
That includes the term "alt right" itself, which, as pointed out in the replies, comes from nazis so they can avoid saying nazis
terminology originating from incel and alt right online communities have resurfaced and entered mainstream media. it is important to stop repeating these words without knowing their origins/meanings
here's a thread of a few common terms you should remove from your vocab:
41
@︎№#ident𝚍 retweeted
terminology originating from incel and alt right online communities have resurfaced and entered mainstream media. it is important to stop repeating these words without knowing their origins/meanings
here's a thread of a few common terms you should remove from your vocab:
@︎№#ident𝚍 retweeted
When your conversation scales as 1/f, you can be sure that you have a nice colleague... Conversation patterns should be like a rustling sea.
science.org/doi/10.1126/scia…
CC: @svalver @PalaeoPhilo @CoelhoPre @BlaiVidiella @niles_eldredge





























