here is my latest blog about the mechanics of a swarm attack and "The Colluder's Dilemma" lesswrong.com/posts/oLHdtvYS…
6
everything is scaled by the norm before it's read. i found scaling of as high as 1200x in gptoss. i call it 'landed writes' lesswrong.com/posts/3KTgeXBf…
“everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy”
this is the scientific revolution here - not just neurons being readable!
2
108
mats needs to start giving per candidate feedback on rejections.
thousands of researchers who don't get in get one bit of feedback every cohort - weak reward signal. easiest way to 5x mats impact imo
🚨 MATS Winter 2027 applications are now open.
Fully-funded, 12-week fellowship for aspiring & established AI alignment, interpretability, security, governance researchers & field-builders
📍 Berkeley/London
📅 Jan 19–Apr 10
💰 $6.4k/mo + $8k - 16k/mo compute
Apply by Sep 6 ↓
9
453
cot is such a good monitor target. llms need to output tokens to interact with the world via tool calling. we cant measure intent but we can prevent harmful tool calls before they are executed
for embodied robots new ideas here are needed
very interesting part on CoT monitoring
> estimation is ~20% compute overhead with a multi level monitoring system
> confirmed they didn't have monitor for eval and training run before
> monitor expected to catch concerning activity within 30min
> safety, security AND (not or) research team get paged and have ~30min to check if it's a false positive, otherwise they stop the run
> sol or higher for the "smart" monitor, the rest is a smaller model (+ likely some hard coded rules?) since it inspects every sampled token
> every inference of astra with tools has a monitor attached to it
1
1
72
our best scientists dont know how huggingface makes money.
maybe it was never meant to make money and this was its fateful purpose
OpenAI is unilaterally pacing the frontier:
- There was a 2-week pause in RL training on latest models intended for deployment.
- "Our largest planned frontier RL run remains on hold".
- A number of Astra-related workloads remain paused.
- Huge compute spend on monitoring: "our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored".
- Changes will be made to the Preparedness Framework to account for the new safeguards and "better reflect the capabilities of future models and the environments in which they operate".
- From Alex Heath's "Sources" newsletter: "executives have not given an estimate for how long the new safety processes could delay the release of Astra"; Mia Glaese (lead on safety and alignment at OpenAI) says: "we are very far from everything running back to normal".
These measures are the result of the Hugging Face incident, the potential designation of Astra as having "Critical" cyber capabilities under the Preparedness Framework, and also "rapid progress in our internal research".
1
169
"our AI lab is focussed on talent and bottlenecked by hiring. this will be key to AI safety"
"no we can not offer any feedback on your application goodbye"
is this a legal thing where they can't give a real reason for not hiring or do they not know they can fix it themselves
Replying to @MicahCarroll
We are hiring!
This may be the best time ever to join: we are incredibly bandwidth bottlenecked and there is a lot of support for almost any impactful misalignment work you can think of. I think we also have a pretty good epistemic environment and a very fun team
Topics: all-things monitoring, misalignment & monitorability assessments, misalignment science, helping set up 3P auditing (e.g. recent Redwood collab), communicating risk externally (system cards/blogs, safety cases, etc)
RSI/misalignment subteam: openai.com/careers/researche…
1
61
I guess to prevent resume goodharting but still. in the old days of faang they had a secret handshake of studying sysdesign problems and learning enough from leetcode to pass. and anyone who was good enough got through eventually
1
11
we don't know why the llm picks any of its words. why would watermarking definitively skew the model to be worse?
the dumbest smart people you know are debating temperature and rngs and skewing probability distributions over semantically similar "words" because they heard about word2vec once and believe "word choice" has no functional purpose beyond a canonical dictionary definition and a cosine similarity in latent space
"isn't hiding a signature in text the same as hiding it in rgb in a pixelated image?", cries the clever-by-half hot-taker
67



















