New research from @WellsLennie (Arrow Research) and @edwardjyoung (@GeodesResearch) showing how knowledge of graders effects cognition learned by LLMs during RL!
We find that when models know information about the grader they are subject to in RL, they generalise to optimising held-out graders they were not trained on.
Although our experiments are in a toy setting, we think it is valuable to consider how these dynamics might effect production RL runs. Without special curation of the pretraining mix, larger models will know more about graders and receive more uplift from reasoning about them.
1
5
105
For the full experimental details and results, including directions for future work, please see our full post: lesswrong.com/posts/qqyxTYss…
If you're interested in working on this kind of problem, you can apply for a full time role here: geodesicresearch.ai/#careers
7
98
We're excited to announce two fellowship streams this winter with @pivotal_org
Come help us with AI Alignment research during some of the most important months in human history. Applications for this round close this Sunday. Link to apply below!👇
In the other, led by Kyle O'Brien, you'll be studying the question: “Since we are concerned about X, is pretraining on X-relevant data making things worse?"
Successful projects here will inform best practices - both in Geodesic as well as frontier pretraining runs.
pivotal-research.org/mentors…
1
3
392
Both streams will be supported by @RadmardPuria
and will tie into the main lines of inquiry being pursued by the Geodesic team at large.
5
241
The other, led by Kyle O'Brien, you'll be studying the question: “Since we are concerned about X, is pretraining on X-relevant data making things worse?"
Successful projects here inform best practices in both Geodesic and frontier pretraining runs.
pivotal-research.org/mentors…
1
8
511
Both streams will be supported by @RadmardPuria and will tie into the main lines of inquiry being pursued by the Geodesic team at large.
4
347
Filtering subversion related information from pretraining is feasible.
We're releasing a research note where we remove information related to monitoring designs and successful subversion strategies in three 30B LLMs.
In collaboration with @redwood_ai 🧵
Authors: Kyle O'Brien, @SpencerKitts, @cam_tice, and @AlekWestover
Thanks for feedback from: @jzlucassen, @ejcgan, @LukasFinnveden, @RyanGreenblatt, @bshlgrs, @tomekkorbak, @edwardjyoung, @RadmardPuria, @DavidDAfrica , and @jammastergirish
1
4
199
See the full research note here: substack.com/home/post/p-219…
And LW blog here: lesswrong.com/posts/HwdXDoCH…
3
121
There's a 𝐠𝐫𝐨𝐰𝐢𝐧𝐠 𝐫𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐠𝐚𝐩 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 𝐬𝐚𝐟𝐞𝐭𝐲 𝐭𝐞𝐚𝐦𝐬 𝐢𝐧𝐬𝐢𝐝𝐞 𝐟𝐫𝐨𝐧𝐭𝐢𝐞𝐫 𝐥𝐚𝐛𝐬 𝐚𝐧𝐝 𝐢𝐧𝐝𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐭 𝐬𝐚𝐟𝐞𝐭𝐲 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬, and closing it is essential to maintaining an ecosystem of impactful safety research.
We've spent months securing a large, multi-year GPU contract. This taught us that 𝐟𝐮𝐧𝐝𝐢𝐧𝐠 𝐢𝐬 𝐧𝐞𝐜𝐞𝐬𝐬𝐚𝐫𝐲 𝐛𝐮𝐭 𝐧𝐨𝐭 𝐬𝐮𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭. We've written up the non-obvious bottlenecks we hit, and what can be done to alleviate them: lesswrong.com/posts/aCGx79eG…
Compute is only half of it. Labs' internal coding agents are likely well ahead of publicly available tools, and safeguards throttle safety work that looks like frontier development; we've hit this on numerous occasions optimising infrastructure. You can't buy your way to parity with token spend.
1
2
126
Our appendix includes the full story of the deal, our procurement strategy and takeaways. Read the full: lesswrong.com/posts/aCGx79eG…
2
114


