Narayanan Sadagopan (@nsdgpn) profile picture

Principal Scientist @ Amazon, Yahoo Labs alum, ex Netflix. Interested in data, machine learning, causality, deep learning, generative models, LLMs

California, USA
Joined March 2016
Photo by Narayanan Sadagopan (@nsdgpn)
Which tokens matter most in on-policy distillation? Excited to share our work at AWS AI Labs with @vijaylingam08 and Yilan Chen. Our Fisher Utility Weighting (FUW) derives token importance from expected loss reduction, connecting Fisher geometry to variance & χ² divergence. 📈 Up to +4.0 accuracy points 🎥 2-min explainer fisher-utility-weighting-opd…
2
2
1
8
335
Narayanan Sadagopan retweeted
Formal math taking off at @PrincetonPLI ! New Goedel-Prover v2 8B model matches 2.5 month old Deepseek V2 prover 671B, but is 80x smaller. Our 32B model much better on all benchmarks (miniF2F, IMO, Putnam). I'm excited/shocked by how much this field has advanced in 6-7 months (since release of Goedel v1). Many technical innovations in v2, e.g., letting the prover generate its own training questions and incorporating compiler feedback. Self-improvement will be v. important to cracking college math (not enough training data!). Kudos to the team, esp. @Yong18850571 @chijinML . Onwards to v3!
(1/4)🚨 Introducing Goedel-Prover V2 🚨 🔥🔥🔥 The strongest open-source theorem prover to date. 🥇 #1 on PutnamBench: Solves 64 problems—with far less compute. 🧠 New SOTA on MiniF2F: * 32B model hits 90.4% at Pass@32, beating DeepSeek-Prover-V2-671B’s 82.4%. * 8B > 671B: Our 8B model matches DeepSeek-671B on MiniF2F. 📚 Leading on MathOlympiadBench (IMO-level problems) * Solves 73 vs 50 over 671B DeepSeek Prover 🔓 Website: blog.goedel-prover.com/ 🔓 Model 32B: huggingface.co/Goedel-LM/Goe… 🔓 Model 8B huggingface.co/Goedel-LM/Goe… 🔓Data and training pipeline will be released soon. Amazing Collaborators: @sangertang1999 @Lyubh22 @__zrrr__ @juihuichung @thomaszhao1998 @pero733858111 @thiiis_user @EmilyJge @JingruoS5931 @wujiayun12 @GesiJiri68334 @davidjesusacu @KaiyuYang4 @hongzhou__lin @YejinChoinka @danqi_chen @prfsanjeevarora @chijinML
20
90
8,469
Interesting work from our team on the shortcomings of reasoning models for instruction following tasks.
Turns out that Chain-of-Thought (CoT) prompting, isn't always a good idea, especially when you just want them to follow instructions precisely. LLMs "thinking" too hard can break simple instructions. 🤔 The Paradox of "Thinking" while CoT helps LLMs tackle complex reasoning tasks, it can significantly mess up their ability to follow specific instructions. This was observed across 15 different LLMs tested on two benchmarks: IFEval (for simple, verifiable rules like "write 400 words") and ComplexBench (for intricate, multi-step instructions). When CoT was used, accuracy in following these instructions often dropped. For example, Llama3-8B-Instruct's accuracy on IFEval fell from 75.2% to 59.0%. ⚙️ So why Reasoning Goes Wrong 📌 When Reasoning Helps: → Formatting/Structure: CoT can assist in producing outputs with correct structures, like valid JSON or markdown. → Lexical Precision: It can help adhere to specific word choices or constraints, like using a rare character multiple times. 📌 When Reasoning Hurts: → Neglecting Simple Rules: Models, while "thinking," might focus too much on the overall content or task, thereby overlooking basic constraints like word limits or exact phrase repetitions. → Adding Unwanted Extras: The reasoning process itself can lead the LLM to introduce extra content—explanations, elaborations—that, while perhaps well-intentioned, end up violating specific instructions (e.g., adding commentary when only a quote was asked for). 📌 Constraint Attention: To quantify this, the researchers proposed a metric called constraint attention. This measures how much attention the model pays to the parts of the input prompt that specify the constraints during output generation. They found that CoT reasoning often diverts the model's attention away from these crucial instruction-relevant tokens. The model "thinks" about the broader task but "forgets" the specific rules it needs to follow for the final answer. 🛠️ Getting LLMs Back on Track: Mitigation Strategies Four strategies were tested to fix this reasoning-induced failure: 📌 In-Context Learning: Providing a few examples of correct instruction-following (with reasoning) within the prompt. This showed only modest improvements, likely due to the limited number of examples that can fit and potential bias from example selection. 📌 Self-Reflection: A two-step process. The model first generates a response with CoT, then reflects on its own reasoning and answer to see if it met all constraints, revising if necessary. This worked well for simpler instructions and larger models but struggled with complex tasks and weaker models. 📌 Self-Selective Reasoning: The model itself decides whether to use CoT for a given instruction. This showed moderate gains, especially on complex tasks, but models often overused reasoning. 📌 Classifier-Selective Reasoning: An external classifier (another trained model) predicts whether using CoT for a specific instruction will improve performance. This was the most consistently effective strategy across both benchmarks, significantly recovering lost performance, though it adds the overhead of training a separate classifier for each target model.
1
1
95
Narayanan Sadagopan retweeted
The Pitfalls of Reasoning for Instruction-Following in LLMs If you're dev using reasoning models, read this one. Lots of great insights and mitigation tactics. Here are my notes:
15
90
6
653
109,341
Narayanan Sadagopan retweeted
This is interesting as a first large diffusion-based LLM. Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different - it doesn't go left to right, but all at once. You start with noise and gradually denoise into a token stream. Most of the image / video generation AI tools actually work this way and use Diffusion, not Autoregression. It's only text (and sometimes audio!) that have resisted. So it's been a bit of a mystery to me and many others why, for some reason, text prefers Autoregression, but images/videos prefer Diffusion. This turns out to be a fairly deep rabbit hole that has to do with the distribution of information and noise and our own perception of them, in these domains. If you look close enough, a lot of interesting connections emerge between the two as well. All that to say that this model has the potential to be different, and possibly showcase new, unique psychology, or new strengths and weaknesses. I encourage people to try it out!
We are excited to introduce Mercury, the first commercial-grade diffusion large language model (dLLM)! dLLMs push the frontier of intelligence and speed with parallel, coarse-to-fine text generation.
371
1,490
185
11,320
947,014
Narayanan Sadagopan retweeted
Meta presents: SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution Achieves 41.0% solve rate on SWE-bench Verified with Llama 3
1
30
3
173
39,767
Narayanan Sadagopan retweeted
🚨Just out Targeted data curation for SFT and RLHF is a significant cost factor 💰for improving LLM performance during post-training. How should you allocate your data annotation budgets between SFT and Preference Data? We ran 1000+ experiments to find out! 1/7
2
33
2
140
16,966
Narayanan Sadagopan retweeted
We can scale up test-time compute with verifiers or without verifiers. It's not obvious which way is better. Turns out we can do some theoretical analysis and show that the use of verifiers is more optimal, and this is supported by experiments!
🚨 RL or distillation/SFT: what to use to train next reasoning model? Which 📈 perf faster as we scale test compute? We answer these in a principled way so you don't have to burn GPUs🔥. 🎯 Ans: RL w/ rewards or verification >> SFT/distillation 😱 arxiv.org/pdf/2502.12118 🧵⤵️
9
31
2
205
29,083
Narayanan Sadagopan retweeted
AI provides a universal framework that leverages data and compute at scale to uncover higher-order patterns Today, @arcinstitute in collaboration with @nvidia releases Evo 2—a fully open source biological foundation model trained on genomes spanning the entire tree of life 🧵
89
555
111
2,585
466,279
Narayanan Sadagopan retweeted
Demystifying Long CoT Reasoning in LLMs arxiv.org/pdf/2502.03373 Reasoning models like R1 / O1 / O3 have gained massive attention, but their training dynamics remain a mystery. We're taking a first deep dive into understanding long CoT reasoning in LLMs! 11 Major Takeaways👇(long threads)
12
225
25
941
187,968
Narayanan Sadagopan retweeted
Scalable-Softmax Is Superior for Attention - Proposes SSMax to process longer context length more effectively - Significantly improves perf in long contexts and key information retrieval
4
38
3
271
44,419
Narayanan Sadagopan retweeted
🚨 Diverse Preference Optimization (DivPO) 🚨 SOTA LLMs have model collapse🫠: they can't generate diverse creative writing or synthetic data 🎨 DivPO trains for both high reward & diversity, vastly improving variety with similar quality. Paper 📝: arxiv.org/abs/2501.18101 🧵below
1
78
6
338
45,405
Narayanan Sadagopan retweeted
Advance in automated math proofs (Lean) from @PrincetonPLI and collaborators. Better than Deepseek-Prover 1.5 --and currently we use only SFT, no RL! (RL version forthcoming). Planned release: dataset of 1.78M formalized proofs to turbocharge open efforts. Key idea: self-improvement loop. Start from usual math Q&A, teach model to "translate" into formal lean proof, and if it checks out, add to training set.
🚀 Introducing Goedel-Prover: A 7B LLM achieving SOTA open-source performance in automated theorem proving! 🔥 ✅ Improving +7% over previous open source SOTA on miniF2F 🏆 Ranking 1st on the PutnamBench Leaderboard 🤖 Solving 1.9X total problems compared to prior works on Lean Workbook [1/n] website: goedel-lm.github.io huggingface: huggingface.co/Goedel-LM/Goe… github: github.com/Goedel-LM/Goedel-… Amazing collaborators: @sangertang1999 (co-first author) @Lyubh22 @wujiayun12 @hongzhou__lin @KaiyuYang4 @JiaLi52524397 @xiamengzhou @danqi_chen @prfsanjeevarora @chijinML
1
41
176
22,694
Narayanan Sadagopan retweeted
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training Shows that: - RL generalizes in rule-based envs, esp. when trained with an outcome-based reward - SFT tends to memorize the training data and struggles to generalize OOD
10
143
12
908
76,463
Narayanan Sadagopan retweeted
These houses are worth Millions. I wonder if they will get treated like the people in North Carolina? This is Malibu - one of the wealthiest affluent places on the entire planet, now it’s being burnt to ashes. Have you ever seen anything like this ever before?
4,374
5,353
586
21,574
2,888,344
Narayanan Sadagopan retweeted
Fantastic Paper from @GoogleDeepMind. Astute RAG enhances LLM performance by resolving conflicts between internal and external knowledge sources. Original Problem 🔍: RAG systems face challenges from imperfect retrieval, introducing irrelevant or misleading information. Knowledge conflicts between LLMs' internal knowledge and external sources undermine RAG effectiveness. ----- Solution in this Paper 🛠️: • Astute RAG: - Adaptive generation of internal LLM knowledge - Iterative source-aware knowledge consolidation - Answer finalization based on information reliability • Addresses knowledge conflicts explicitly • Combines internal and external knowledge effectively ----- Key Insights from this Paper 💡: • Imperfect retrieval is prevalent in real-world RAG (70% of retrieved passages lack direct answers) • Knowledge conflicts exist in 19.2% of cases • LLM internal knowledge and external sources have distinct advantages • Effective combination of internal and external knowledge is crucial for reliable RAG ----- Results 📊: • Outperforms baselines across datasets: - 6.85% relative improvement on Claude - 4.13% improvement on Gemini • Only method matching/exceeding No-RAG performance in worst-case scenarios • Resolves 80% of knowledge conflicts correctly • Improves performance even when neither knowledge source alone is correct
6
120
4
713
63,065
Narayanan Sadagopan retweeted
I’m pleased to announce our work which studies complexity phase transitions in neural networks! We track the Kolmogorov complexity of networks as they “grok”, and find a characteristic rise and fall of complexity, corresponding to memorization followed by generalization. 🧵
31
152
26
1,161
208,971
🚨 Our paper "AXCEL: Automated eXplainable Consistency Evaluation using LLMs" is accepted at #EMNLP2024! 🎉 AXCEL improves consistency detection by 8.7% (summarization), 6.2% (free-text), & 29.4% (data-to-text), offering explainable results! 🔍📊 #LLMs arxiv.org/abs/2409.16984
1
66
Narayanan Sadagopan retweeted
A barrier to faster progress in generative AI is evaluations (evals), particularly of custom AI applications that generate free-form text. Let’s say you have a multi-agent research system that includes a researcher agent and a writer agent. Would adding a fact-checking agent improve the results? If we can’t efficiently evaluate the impact of such changes, it’s hard to know which changes to keep. For evaluating general-purpose foundation models such as large language models (LLMs) — which are trained to respond to a large variety of prompts — we have standardized tests like MMLU (multiple-choice questions that cover 57 disciplines like math, philosophy, and medicine) and HumanEval (testing code generation); the LMSYS Chatbot arena, which pits two LLMs’ responses against each other and asks a human to judge which response is superior; and large-scale benchmarking like HELM. These evaluation tools took considerable effort to build, and they are invaluable for giving LLM users a sense of different models' relative performance. Nonetheless, they have limitations: For example, leakage of benchmarks datasets’ questions and answers into training data is a constant worry, and human preference for certain answers does not mean those answers are more accurate. In contrast, our current options for evaluating specific applications built using LLMs are far more limited. Here, I see two major types of applications. - For applications designed to deliver unambiguous, right-or-wrong responses, we have reasonable options. Let’s say we want an LLM to read a resume and extract the candidate's most recent job title, or read a customer email and route it to the right department. We can create a test set that comprises ground-truth labeled examples with the right responses, and measure the percentage of times the LLM generates the right output. The main bottleneck is creating the labeled test set, which is expensive but surmountable. - But many LLM-based applications generate free-text output with no single right response. For example, if we ask an LLM to summarize customer emails, there’s a multitude of possible good (and bad) responses. The same holds for an agentic system to do web research and write an article about a topic, or a RAG system for answering questions. It’s impractical to hire an army of human experts to read the LLM’s outputs every time we tweak the algorithm and evaluate if the answers have improved — we need an automated way to test the outputs. Thus, many teams use an advanced language model to evaluate outputs. In the customer email summarization example, we might design an evaluation rubric (scoring criteria) for what makes a good summary. Given an email summary generated by our system, we might prompt an advanced LLM to read it and score it according to our rubric. I’ve found that the results of such a procedure, while better than nothing, can also be noisy — sometimes too noisy to reliably tell me if the way I’ve tweaked an algorithm is good or bad. The cost of running evals poses an additional challenge. Let’s say you’re using an LLM that costs $10 per million input tokens, and a typical query has 1000 tokens. Each user query therefore costs only $0.01. However, if you iteratively work to improve your algorithm based on 1000 test examples, and if in a single day you evaluate 20 ideas, then your cost will be 20*1000*0.01 = $200. For many projects I’ve worked on, the development costs were fairly negligible until we started doing evals, whereupon the costs suddenly increased. (If the product turned out to be successful, then costs increased even more at deployment, but that was something we were happy to see!) In addition to the dollar cost, evals also have a significant time cost. Running evals on 1000 examples might take tens of minutes or even hours. Time spent waiting for eval jobs to finish also slows down the speed with which we can experiment and iterate over new ideas. Previously I wrote that fast, inexpensive token generation is critical for agentic workflows. This will also be useful for evals, which involve nested for-loops that iterate over a test set and different model/hyperparameter/prompt choices and therefore consume large numbers of tokens. Despite the limitations of today's eval methodologies, I’m optimistic that our community will invent better techniques (maybe involving agentic workflows like reflection?) for getting LLMs to evaluate such output. If you’re a developer or researcher and have ideas along these lines, I hope you’ll keep working on them and consider open sourcing or publishing your findings! [Original text: deeplearning.ai/the-batch/is… ]
51
155
26
868
187,271