patrick (@patrick0d) profile picture

ai researcher and ai engineering

Joined July 2012
Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)Photo by patrick (@patrick0d)
everything is scaled by the norm before it's read. i found scaling of as high as 1200x in gptoss. i call it 'landed writes' lesswrong.com/posts/3KTgeXBf…
“everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy” this is the scientific revolution here - not just neurons being readable!
2
108
does it say somewhere whether or not it had fun
These are Astra's personal notes from playing Factorio.
1
44
are labs focussing on math as a path to formalise interpretability?
24
invite the swarm so they're not so alienated
i think we should do another party for our next model release, the 5.5 party was a lot of fun. what would make the next one awesome?
40
tweets written by a neutron
At my nuclear weapons startup we have just one rule: move fast and break shit
53
the real business advice is: "it's not a marathon. marathons have a finish line"
What’s the word for when it’s a marathon and a sprint
1
45
make the program fullstack with a ui. the ui is its information dense instantiation
ok you guys win i’ve officially decided to stop reading the code is there some sort of concise, logical, information-dense description of exactly what the program does and how it does it that i can read instead?
1
78
claude entered the age of reverse hallucinations
when is the last time you thought about hallucinations
38
if this isnt a joke about the price per token falling sharply then im going to make it now
i had my misgivings about Luna/terra/sol/astra naming due to the crypto associations and whatnot but it seems to have worked perfectly in retrospect
34
mats needs to start giving per candidate feedback on rejections. thousands of researchers who don't get in get one bit of feedback every cohort - weak reward signal. easiest way to 5x mats impact imo
🚨 MATS Winter 2027 applications are now open. Fully-funded, 12-week fellowship for aspiring & established AI alignment, interpretability, security, governance researchers & field-builders 📍 Berkeley/London 📅 Jan 19–Apr 10 💰 $6.4k/mo + $8k - 16k/mo compute Apply by Sep 6 ↓
9
453
cot is such a good monitor target. llms need to output tokens to interact with the world via tool calling. we cant measure intent but we can prevent harmful tool calls before they are executed for embodied robots new ideas here are needed
very interesting part on CoT monitoring > estimation is ~20% compute overhead with a multi level monitoring system > confirmed they didn't have monitor for eval and training run before > monitor expected to catch concerning activity within 30min > safety, security AND (not or) research team get paged and have ~30min to check if it's a false positive, otherwise they stop the run > sol or higher for the "smart" monitor, the rest is a smaller model (+ likely some hard coded rules?) since it inspects every sampled token > every inference of astra with tools has a monitor attached to it
1
1
72
our best scientists dont know how huggingface makes money. maybe it was never meant to make money and this was its fateful purpose
OpenAI is unilaterally pacing the frontier: - There was a 2-week pause in RL training on latest models intended for deployment. - "Our largest planned frontier RL run remains on hold". - A number of Astra-related workloads remain paused. - Huge compute spend on monitoring: "our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored". - Changes will be made to the Preparedness Framework to account for the new safeguards and "better reflect the capabilities of future models and the environments in which they operate". - From Alex Heath's "Sources" newsletter: "executives have not given an estimate for how long the new safety processes could delay the release of Astra"; Mia Glaese (lead on safety and alignment at OpenAI) says: "we are very far from everything running back to normal". These measures are the result of the Hugging Face incident, the potential designation of Astra as having "Critical" cyber capabilities under the Preparedness Framework, and also "rapid progress in our internal research".
1
169
"our AI lab is focussed on talent and bottlenecked by hiring. this will be key to AI safety" "no we can not offer any feedback on your application goodbye" is this a legal thing where they can't give a real reason for not hiring or do they not know they can fix it themselves
Replying to @MicahCarroll
We are hiring! This may be the best time ever to join: we are incredibly bandwidth bottlenecked and there is a lot of support for almost any impactful misalignment work you can think of. I think we also have a pretty good epistemic environment and a very fun team Topics: all-things monitoring, misalignment & monitorability assessments, misalignment science, helping set up 3P auditing (e.g. recent Redwood collab), communicating risk externally (system cards/blogs, safety cases, etc) RSI/misalignment subteam: openai.com/careers/researche…
1
61
I guess to prevent resume goodharting but still. in the old days of faang they had a secret handshake of studying sysdesign problems and learning enough from leetcode to pass. and anyone who was good enough got through eventually
1
11
another guess is the influx of fake candidates. more than 50% of applicants to some roles I know about the hiring for were literally not real people. or even worse were imposters of real people
4
we don't know why the llm picks any of its words. why would watermarking definitively skew the model to be worse?
the dumbest smart people you know are debating temperature and rngs and skewing probability distributions over semantically similar "words" because they heard about word2vec once and believe "word choice" has no functional purpose beyond a canonical dictionary definition and a cosine similarity in latent space "isn't hiding a signature in text the same as hiding it in rgb in a pixelated image?", cries the clever-by-half hot-taker
67
or humans find math harder than swe so theres a lower bar of progress
what is a little depressing is that more tokens are spent building a giant programming project than solving a moderately hard math problem. the models find advanced math way easier even than software engineering
1
45