Daniel Nichols retweeted
New mini experiment + blogpost + trajectories!
tldr; we boost performance of RLM(GPT-5.2) to double the best performing number (38.7% --> 65.6%) on LongCoT-mini without any training! An example of the mismanaged geniuses hypothesis (MGH) we (@zli11010, @lateinteraction) proposed earlier this month.
The LongCoT benchmark showed that frontier LMs and RLMs struggled to solve difficult compositional reasoning tasks. The paper generally attributes this to the RLMs inability to perform task decomposition, but we argue this is more our fault in how we prompt them; this capability is fully available to GPT-5.2 with an RLM harness!
Building on @raw_works's insightful blogpost and @sumeetrm / @CharlieLondon02 et al.'s incredibly useful benchmark, where they originally found RLMs to be incapable of solving the MATH and CS splits altogether. We did not train anything since the release of the initial benchmark.
To be fully transparent, these results are not meant to be added to their leaderboard either; benchmarks measure isolated capabilities, and we focus on showing (through different, rather specific prompting) that the capabilities required to solve these tasks are available to the models without additional training! It also has implications about how we would go about training these systems. Full blog below, it's a nice read :)
Daniel Nichols retweeted
LongCoT is adding two new leaderboards! Due to the interest in agents (particularly RLMs), we’re adding a “Restricted Harness” and an “Open Harness” leaderboard.
GPT 5.2 RLM from our paper is SOTA on “Open Harness” at 25.12%. We expect tool-use SOTA to exceed this very soon!
On “Open Harness”, we allow all tool-use and code execution. On “Restricted Harness”, models may manage context, call subagents, etc, but may not write specific solver code (e.g. writing a BlocksWorld or Sudoku solver). We’re particularly excited about this leaderboard, as it allows agents to do their own context management, while sticking to LongCoT’s goal of testing models’ intrinsic reasoning capabilities.
Daniel Nichols retweeted
We’re releasing LongCoT, an incredibly hard benchmark to measure long-horizon reasoning capabilities over tens to hundreds of thousands of tokens.
LongCoT consists of 2.5K questions across chemistry, math, chess, logic, and computer science. Frontier models score less than 10%🧵
Daniel Nichols retweeted
Meet our 2025 Fernbach Fellow, Daniel Nichols! His research explores how AI can be leveraged to maximize the efficiency & impact of HPC resources.
computing.llnl.gov/about/peo…
Daniel Nichols retweeted
🚨 New paper alert 🚨
Can Large Language Models Write Parallel Code?
@DanielNichols10 and @jhdavis_josh have conducted a comprehensive study of several state-of-the-art open-source code and GPT models for many parallel libraries/runtimes. #HPC #AI #DL
arxiv.org/abs/2401.12554
Daniel Nichols retweeted
2nd New Paper Alert! PSSG student @DanielNichols10 is tackling the problem of performance variability. Joint work w/ Aniruddha Marathe, Kathleen Shoga, & @tgamblin of @Livermore_Comp to be presented @IPDPS 2022.
pssg.cs.umd.edu/assets/paper…
Interesting details about the paper 👇🧵
Thank you @thecollectionnc for spending 5,103 minutes with me this year on @Spotify. You are my #1. #2020Wrapped open.spotify.com/artist/3sva…
1
1
Daniel Nichols retweeted
This Admiral Schofield poster had everybody out of their seat 😨 nitter.cf/BleacherReport/status/…















