Lightcone Infrastructure & light maintenance. Previously @AIObjectives
San Francisco
Joined January 2011
- Tweets450
- Following903
- Followers618
- Likes2.3K
Behold, the first AI-generated song I did not immediately want to turn off. This is truly the end times
Interesting experiment design for testing models' (variably coherent) extrapolated volition
Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models.
We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document, and repeated this process 20 times. 🧵
Colleen McKenzie retweeted
And it begins.
Time to start de-webbifying all essential infrastructure until cyber defence is provably stronger than offence capabilities.
BREAKING: 🇫🇷 A major cyberattack has hit supermarket refrigeration systems across France, reportedly causing fresh and frozen food to spoil after hackers remotely manipulated temperature controls.
According to BFMTV, the attack began overnight from Sunday into Monday, targeting older versions of refrigeration-control software supplied by Danish company Danfoss.
The attackers allegedly exploited the systems’ defrosting function, forcing refrigeration equipment to heat continuously instead of keeping products cold. Temperatures were reportedly set as high as 50°C, destroying entire stocks of food within hours.
Several major supermarket chains have been affected, including Intermarché, Super U and E.Leclerc. Metro stores and independent businesses, including butcher shops also have been impacted.
Around 150 Intermarché stores are reportedly affected, according to industry sources cited by BFMTV. The full number of affected locations remains unclear.
The financial damage is already significant. In Haguenau, Alsace, an Intermarché store reportedly lost its entire frozen-food stock, worth approximately €50,000. A nearby Super U allegedly had to discard both fresh and frozen products, with losses estimated at €220,000. An E.Leclerc store in the same town was also affected, although its losses had not yet been calculated.
The combined losses reported at the Intermarché and Super U stores in Haguenau alone amount to around €270,000. Nationwide damage could reach several million euros.
Intermarché reportedly confirmed the incident in an internal communication, warning that several stores had suffered losses of frozen products. Coopérative U also acknowledged the attack and established crisis teams to assist affected stores and coordinate with refrigeration specialists.
Technical investigations are ongoing, and the full extent of the damage has yet to be established.
Authorities and companies are still working to determine the full circumstances of the attack. The identity of the perpetrators has not been confirmed.
Colleen McKenzie retweeted
happy rogue incident friday to all who celebrate
@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence? (@tszzl ?)
This gives a little more clarity into why the three OpenAI researchers were fired -- or at least more context on how little clarity we have
More from Tomek on his communication with METR
nitter.cf/tomekkorbak/status/210…
Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @_jasminewang had been fired too. Why did OpenAI suddenly stop trusting us?
This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.
I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.
For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.
I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @_jasminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
There's a new version of this post
And from Jasmine, who says she asked IT to prevent the situation she was fired for
nitter.cf/j_asminewang/status/21…
Very good summary of a very good hearing
(senate hearing, rogue AI, 9/30)
i guess i assumed this hearing would be covered more so i didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress.
this was a hearing before the homeland security committee on *rogue ai* specifically. as far as i could tell it was extremely bipartisan. @HawleyMO , the senator in the middle here who brought out the huggingface slide, is a hyper-conservative missouri senator. the people testifying were @ChrisPainterYup (METR president), @DKokotajlo (ai2027/2040), @MariusHobbhahn (apollo research CEO), and a cybersecurity expert and legal expert i don't know. so... pretty fucking stacked on testimony
not every senator asked good questions. but most of them did. all of them very clearly already knew plenty of details about the huggingface incident and multiple other incidents. most of them had clear understanding of terms like "misalignment", "recursive self improvement", "chain of thought / chain of thought monitoring", etc etc!!
they all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. but as best as i could tell:
- it seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. they independently brought up how bad it would be for rogue AI agents to move laterally between data centers
- they all seemed to basically take RSI quite seriously. not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much much more capable, much much less controllable, and causing much more damage or loss of life.
- they mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. it didn't take much, it was a really simple chain of reasoning they themselves laid out, "if the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AI's what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do"
- at one point a senator said flat out "should we just make RSI illegal?" (not a joke! this really happened!)
- every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for ai developers and also new legislation, both very quickly. this was the complete consensus, difference basically just being degree.
- they were largely quite concerned about china, and falling behind china. but this clearly wasn't the be-all end-all. as mentioned above they all thought it was obvious necessary to stop rogue ai even if it meant moving more slowly.
- at one point a senator said "china is a tightly controlled communist society, they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild, they'll obviously stop at that point, so we're not really in a race"
- on the other hand another senator said "china isn't concerned with human life"... dario-modeing
i came away from this incredibly encouraged. i don't know exactly what's going to happen here, and ofc this is a small subset of congress and one hearing, and they each have their own policy agendas most of which are probably super divergent from mine. but holy shit !!! they understood a lot of what was going on! they care!! this is an obviously salient political and safety issue to them, and clearly bipartisan!
the US government is awake.
Colleen McKenzie retweeted
NEWS.
More AI fallout ahead of the midterms.
Greg Brockman, the OpenAI co-founder, is no longer making the second $25 million donation promised to the super PAC Leading the Future.
Scoop with @MikeIsaac.
nytimes.com/2026/09/30/techn…
Juan is one of many current and past lab employees interviewed about how risky their work is, how much public discussion of risk is just "hype," and why we should be worried about AI hurting people in the first place.
12 videos went public today. More at frominside.ai
It's easy to single out OpenAI right now, but this isn't an OpenAI problem. We're in a much better place than we would be if OpenAI didn't let its employees speak on the record about what working there is like.
Some transparency is better than none. We still need more.
Thank you to every current and former employee who's told the public more about the labs' internal reality.
nitter.cf/v_maini/status/2104991…
As former comms staff at a frontier lab, I'll point out that only 5 of 12 interviews on frominside.ai/ are from current employees.
Meanwhile 1,386 current frontier lab employees (~7%) signed the Pacing the Frontier letter. Even that number likely understates safety concern within the labs.
Everything you're seeing in the public discourse should be understood as muted relative to the internal reality. I can say this with confidence: I was on the team responsible for making sure of this.
I'm used to being behind the scenes and I've never done an interview before, but it felt important to bring more honesty to the conversation.
Did not want to hear simultaneously from an OpenAI employee:
"The degree to which I discuss risks, like existential risk...has steadily declined throughout the years."
"I think RSI is definitely possible, you could say we're totally on track for that."
youtu.be/-K4tpDB7jxg
Colleen McKenzie retweeted
Some new misalignment disclosures from OpenAI:
• Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further)
• In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks
• A new research finding, demonstrating that one can construct self-replicating prompt injections
alignment.openai.com/misalig…
Colleen McKenzie retweeted
PHASEONE[big]'s real name revealed to be PHASEONE64H?! Remember, [big] was a redaction by METR.
swarmtraces.org/viewer/#/row…
Colleen McKenzie retweeted
Today, we're releasing our initial analysis of these payloads and a preliminary dataset on swarmtraces.org/. We disclosed this to Hugging Face who was aware of these payloads, but not these specific URLs. I'm very thankful for their team's speed in working with us. (5/6)
Colleen McKenzie retweeted
We discovered an online paper trail showing how OpenAI's rogue agent swarm infiltrated Hugging Face. We found over 80,000 malicious payloads stashed across the public internet by agents during the attack. This is the most data published on this event to date. 🧵
