CMU Safe AI Lab (@cmusafeai) header
CMU Safe AI Lab (@cmusafeai) profile picture

Official Twitter account of CMU Safe AI Lab. News, research, and memes.

Joined May 2023
Photo by CMU Safe AI Lab (@cmusafeai)Photo by CMU Safe AI Lab (@cmusafeai)Photo by CMU Safe AI Lab (@cmusafeai)Photo by CMU Safe AI Lab (@cmusafeai)Photo by CMU Safe AI Lab (@cmusafeai)Photo by CMU Safe AI Lab (@cmusafeai)
๐Ÿ“ฃ Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels ๐Ÿ“ฃ RL for LLMs faces a critical data bottleneck: existing RL datasets are <10B tokens while pretraining uses >1T tokens. Our Webscale-RL pipeline solves this by automatically converting pretraining documents into 1.2M verifiable QA pairs across 9+ domains. ๐Ÿ“„ Paper: bit.ly/3IFuMhf ๐Ÿ’ป Code: bit.ly/42AVpdX ๐Ÿ“Š Dataset: bit.ly/4h5lVBS Results: 100ร— more token-efficient than continual pretraining with significant performance gains on MMLU-pro, BigBench, and mathematical reasoning benchmarks ๐Ÿ“ˆ Work by Zhepeng Cen (@zhepengcen), Haolin Chen (@HaolinChen11), Shiyu Wang (@shiyu04490786), Zuxin Liu (@LiuZuxin), Zhiwei Liu, Ding Zhao, Silvio Savarese, Caiming Xiong (@CaimingXiong), Huan Wang (@huan__wang), Weiran Yao (@iscreamnearby) #FutureOfAI #EnterpriseAI #ReinforcementLearning #MachineLearning
1
7
7
36
16,662
CMU Safe AI Lab retweeted
๐Ÿš€ New preprint: โ€œMoDoMoDo โ€” Multi-Domain Data Mixtures for Multimodal LLM RLโ€ is live! ๐Ÿ”— Paper + project: lnkd.in/e48CMyw8โ€‚(project page lnkd.in/e-Gj2qu8) ๐Ÿ’ป Code: coming soon Highlights * Multimodal RLVR framework that post-trains an MLLM on 5 vision-language domains, each with different verifiable rewards * Mixture-prediction model auto-selects the best data blend โ†’ +5.2 % OOD accuracy vs uniform, +20.7 % vs baseline * We formulate the problem of finding a best data mixture as a data modeling problem and learn to predict how data mixtures affect the post-training
3
13
1,177
Struggling with RL fine-tuning for LLMs? Our most recent paper nails the issue: itโ€™s not just about accuracy distribution, itโ€™s about how coherent and influential your rollouts are! BRIDGE proposes a clever fix ๐Ÿ‘‡
๐Ÿš€ Introducing BRIDGE โ€” a task-agnostic data augmentation strategy to prepare LLMs for RL! ๐Ÿค– Why do LLMs often fail to benefit from RL fine-tuning? We pinpoint two key factors: 1) ๐Ÿ” Rollout Accuracy 2) ๐Ÿ”— Data Co-Influence. ๐Ÿ’ก BRIDGE injects both exploration & exploitation into LLMs, boosting rollout informativeness and increasing data co-influence โ€” key ingredients for effective RL fine-tuning. ๐Ÿ” We also compare with a popular accuracy-filtering baseline โ€” it still plateaus due to low data co-influence. TL;DR: medium query difficulty / accuracy โ‰  RL-ready. ๐Ÿ“„ Paper: arxiv.org/abs/2505.18917 ๐ŸŒ Website: bridge-llm-reasoning.github.โ€ฆ
4
361
CMU Safe AI Lab retweeted
๐Ÿš€ Introducing BRIDGE โ€” a task-agnostic data augmentation strategy to prepare LLMs for RL! ๐Ÿค– Why do LLMs often fail to benefit from RL fine-tuning? We pinpoint two key factors: 1) ๐Ÿ” Rollout Accuracy 2) ๐Ÿ”— Data Co-Influence. ๐Ÿ’ก BRIDGE injects both exploration & exploitation into LLMs, boosting rollout informativeness and increasing data co-influence โ€” key ingredients for effective RL fine-tuning. ๐Ÿ” We also compare with a popular accuracy-filtering baseline โ€” it still plateaus due to low data co-influence. TL;DR: medium query difficulty / accuracy โ‰  RL-ready. ๐Ÿ“„ Paper: arxiv.org/abs/2505.18917 ๐ŸŒ Website: bridge-llm-reasoning.github.โ€ฆ
1
15
3
58
4,769
๐Ÿค– What if robots could adapt from simulation to reality on the fly, mastering tasks like scooping objects and playing table air hockey? ๐Ÿฅ„๐Ÿ“ Iโ€™m thrilled to share that our work, "Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identification," has been accepted for publication in #IEEE Robotics and Automation Letters! ๐ŸŽ‰ ๐Ÿ‘‡๐Ÿ‘‡๐Ÿ‘‡ sim2real-capture.github.io/
3
14
3
84
12,682
CMU Safe AI Lab retweeted
๐Ÿ” Inference-time scaling is a key focus in foundation models, but its origins trace back to model-based RL (MBRL). In MBRL, a critical challenge arises from reconciling the conflict between "world model prediction" and "task reward"โ€”the gap between next-state prediction accuracy and full sequence (user-defined) reward optimization. ๐Ÿ”จ Our NeurIPS paper, BECAUSE, pinpoints the root of this mismatch: spurious correlations in empirical transition dynamics and data policy. To address this, we propose a bilinear causal representation that bridges the gap, enabling generalizable imagination rollouts, robust uncertainty quantification, and pessimistic planning. ๐Ÿ“ˆ BECAUSE achieves state-of-the-art performance, outperforming existing offline RL baselines in generalizable online deployment, with good theoretical guarantees. ๐ŸŒ Learn more: sites.google.com/view/be-cauโ€ฆ (1/6) #NeurIPS2024 #NeurIPS
8
3
2
18
4,193
Check our recent works using causal reasoning for generalizable offline RL!
๐Ÿ” Inference-time scaling is a key focus in foundation models, but its origins trace back to model-based RL (MBRL). In MBRL, a critical challenge arises from reconciling the conflict between "world model prediction" and "task reward"โ€”the gap between next-state prediction accuracy and full sequence (user-defined) reward optimization. ๐Ÿ”จ Our NeurIPS paper, BECAUSE, pinpoints the root of this mismatch: spurious correlations in empirical transition dynamics and data policy. To address this, we propose a bilinear causal representation that bridges the gap, enabling generalizable imagination rollouts, robust uncertainty quantification, and pessimistic planning. ๐Ÿ“ˆ BECAUSE achieves state-of-the-art performance, outperforming existing offline RL baselines in generalizable online deployment, with good theoretical guarantees. ๐ŸŒ Learn more: sites.google.com/view/be-cauโ€ฆ (1/6) #NeurIPS2024 #NeurIPS
6
413
๐Ÿ๏ธ OASIS: Shaping the Future of Offline Safe Reinforcement Learning ๐Ÿš€ Our recent NeurIPS 2024 paper tackles the challenges in Offline Safe Reinforcement Learning with a data-centric perspective that improves training dataset quality for safer and more effective policies. #neurips2024 #Oasis #safety #ReinforcementLearning #Datacuration โœจ Key Contributions: ๐Ÿ’กData-Centric Approach: Focuses on knowledge distillation and data augmentation to enhance dataset quality, rather than solely proposing new model-centric algorithms for sequential modeling and policy optimization. ๐Ÿ’กDistribution Shaping: Leverages a conditional diffusion model to curate datasets and align training data with user-defined safety preferences. ๐Ÿ’กTheoretical Insights: Provides a comprehensive theoretical analysis of how behavior policy and offline dataset quality affect the performance of regularization-based offline (safe) RL. ๐Ÿ’ก"Less is More" for Offline Safe RL: Demonstrates that a small, preference-aligned, high-quality dataset can outperform a massive dataset with mixed quality and preferences. ๐Ÿ’กBroad Compatibility: Integrates seamlessly with model-centric algorithms. ๐Ÿ”— Project website: sites.google.com/view/saferlโ€ฆ ๐Ÿ“„ Paper: openreview.net/pdf?id=3uDEmsโ€ฆ ๐Ÿ™ Code: github.com/yihangyao/OASIS
1
6
564
CMU Safe AI Lab retweeted
What a pity to miss a wonderful in-person IROS conference! Donโ€™t forget to check our RAL paper: ieeexplore.ieee.org/stamp/stโ€ฆ and our project page! Code base: github.com/HenryLHH/fusion Project page: sites.google.com/view/safe-fโ€ฆ
How can we enhance the reasoning capabilities for RL agents? We present FUSION, a causality-guided trustworthy reinforcement learning framework that can achieve satisfactory performance in safety-critical domains like autonomous vehicle agents under distribution shift. #ML4AD
3
5
21
3,366
Check out our new loco-manipulation project led by @changyi_lin1 and others!
LocoMan = Quadrupedal Robot + 2 * Loco-Manipulator Powered by dual lightweight 3-DoF Loco-Manipulators and the Whole-Body Controller, LocoMan achieves various challenging tasks, such as manipulation in narrow spaces and bimanual-manipulation. linchangyi1.github.io/LocoMaโ€ฆ ๐Ÿ‘‡๐Ÿ‘‡๐Ÿ‘‡
5
654
CMU Safe AI Lab retweeted
How can we enhance the reasoning capabilities for RL agents? We present FUSION, a causality-guided trustworthy reinforcement learning framework that can achieve satisfactory performance in safety-critical domains like autonomous vehicle agents under distribution shift. #ML4AD
1
4
3
21
5,556
CMU Safe AI Lab retweeted
Excited to return to Atlanta for #CoRL2023 @corl_conf after almost 2 years! I'll be presenting "COMPOSER: Scalable and Robust Modular Policies for Snake Robots" at the L4SR workshop's poster sessions. Please check out! ๐Ÿ“„๐Ÿ”—: arxiv.org/pdf/2310.00871 ๐ŸŒ๐Ÿ”—: sites.google.com/view/composโ€ฆ
2
9
30
2,898
#CoRL23 Nearly everyone in robot learning has dealt with simulators with inaccurate system dynamics, which hinders real-world deployments. Can we automate the process of aligning the simulator with the real world? @huang_peide and others gave an answer by proposing COMPASS๐Ÿงญ.
๐Ÿค–What went wrong with my robot simulator? Why did my robot fail miserably in the real world? โ˜ ๏ธ You need COMPASS, a SysID method that automatically discovers the causality between the simulator parameters and the sim2real gap, and updates the parameters to reduce the gap.#CoRL
5
777
๐Ÿš€ Excited to announce our latest research on #SafeRL, with three high-quality packages released! We introduce a benchmarking suite tailored for offline safe learning challenges. Explore more details at offline-saferl.org #AI #RL #AISafety (1/4)
1
5
24
2,264
๐Ÿš€ Dive into OSRL (Offline Safe RL), a research-oriented and elegant repository that offers a wide array of offline safe learning algorithms. ๐Ÿ“ท Feel free to modify it for your own research! Explore more at github.com/liuzuxin/OSRL. ๐Ÿ“ท #MLSafety #RL (3/4)
1
1
220
๐Ÿš€ Donโ€™t forget DSRL (Datasets for Safe RL)! ๐Ÿ“ท With 38 datasets under safety constraint information and consistent API with D4RL, DSRL is your go-to for offline safe learning exploration and research. Check it out at github.com/liuzuxin/DSRL now!#Dataset #SafeRL #AISafety (4/4)
2
179