CS Grad student @ BAIR, UC Berkeley
Berkeley, CA
Joined June 2024
- Tweets62
- Following140
- Followers173
- Likes42
Joseph Suh retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Joseph Suh retweeted
We wrote a blog post on how we achieved such great performance on DSV41-Flash. Our results are also verified in the @SemiAnalysis_ AgentX benchmark.
vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64.
DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.
Joseph Suh retweeted
User simulators are trained to talk like users. But users don’t just talk, they judge.
We introduce simulators that model both: what users say and what they think of the assistant.
Joseph Suh retweeted
Second piece of news, I’m looking to recruit multiple PhD students to work on human-centered AI/alignment/LLM social reasoning this cycle @UTiSchool
While I’m sadly missing #COLM2026 I'll be heading to #EMNLP2026 🇭🇺
Whether you'll be there or not, reach out if you're interested!
Joseph Suh retweeted
Hello my 406 twitter followers. Hopping on to share two things:
1. I’ve started a CS PhD at Stanford! For the love of the game ofc.
2. I’m releasing impersona-env, a proof of concept to train personalized assistants by building an RL environment out of user models. Your assistant learns to help you by repeated asking a simulated version of you questions, probing its internal latent thinking and feedback, and updating its priors accordingly. I think this unlocks a new dimension to personalization beyond discrete fact recall. My personal assistant model now understands nuances of my preferences that even I didn’t know about.
Full blog: benshi34.github.io/#/blog/im…
It’s still a proof of concept and there’s still a ton of things to figure out in this space. It’s unclear to me how to build user model evals that generalize across users and downstream applications, how to best represent longitudinal personal data at training time // inference time, the interplay between population level and individual level preferences, the immense questions surrounding privacy and hillclimbing in decentralized data access regime… if you want to chat on anything in this area please dm me!
Joseph Suh retweeted
I'm at COLM this week, presenting:
- LLMs Corrupt Your Documents When You Delegate. At today (Wednesday's) afternoon poster 4.30-6.30. Come check out our very (!) colorful poster.
- Giving a fun talk on mostly ongoing work at the Workshop on Lifelong Agents early Friday morning.
Old and new friends, let's chat!
WildChat, LMSYS, ShareChat… each human-LLM chat dataset aims to capture "real-world" AI use. But do they paint the same picture?
Not quite. From user messages alone, a classifier can name a chat's source dataset, and user models trained on them inherit these signatures 🧵👇
Takeaway 🥡: no single conversation dataset stands in for "real-world" users. Document how the data was collected, and test any conclusions across datasets.
Paper: arxiv.org/abs/2610.05534
Code: github.com/JosephJeesungSuh/…
Huge thanks to my amazing PI @serinachang5
Why does this matter? Each dataset is a "slice" of human-LLM interaction, and the slices differ in ways that taxonomy summaries miss. A conclusion drawn from one slice, about users, user models, or assistants, may not hold for another.
More patterns:
- Each user model fits its own dataset best; the ranking of the rest shifts with the test set
- Dataset separability strongly correlates with the cross-dataset transfer gap
- Classifier-guided data selection beats random sampling at the same budget
This changes interactive evaluation result of assistants.
On the gsm8k problems, Qwen2.5-14B's success rate ranges 9.6-26.8% (2.8×) across user models.
Does it beat Llama3.1-8B? With the WildChat-1M user model, they're nearly tied (9.6% vs 10.0%). With ShareGPT: 24.8% vs 14.4%.
Do signatures carry over into user models? Yes.
We train one user model per dataset, give them identical intents, and pair them with the same assistant. A classifier still identifies each user model's training dataset with 63-92% accuracy (chance is 50%).
Is it just topics? Mostly not.
We matched datasets on human-designed taxonomy categories (what users discuss, what they ask the assistant to do, how they follow up) plus turn counts and message length. Accuracy dropped by only 3-6 points, staying at 75-83% (chance is 50%).
🔍 Key finding: user text alone gives away the source.
- 7-way: 53.3% accuracy (chance is 14.3%)
- Even related pairs (two WildChat releases, two Chatbot Arena collections, two shared-chat collections): 62.8 – 86.8% (chance is 50%)
Our approach: bring the classic "Name That Dataset!" experiment (Torralba & Efros, 2011) from computer vision to human-LLM chats.
We train a classifier to predict which dataset a chat sample came from, with dataset-specific markers removed and all assistant responses masked.
Why does the dataset choice matter?
These datasets play two roles:
1. Evidence: they shape what we learn about how people use AI
2. Training data: user models (aka user simulators) trained on them are used to train and evaluate AI assistants
Joseph Suh retweeted
🚢Marin 535B-A23B training has crossed the halfway mark! Some observations and learnings so far:
Joseph Suh retweeted
My problem with Astra reviews is that they focus almost entirely on intra-paper details: experimental rigor, missing ablations, and whether every local claim is airtight.
But good reviewing also needs an inter-paper view: what can we learn from this work, how does it change how we think about the problem, and what future research can it enable?
Research is more than a checklist.
If ICLR gets flooded with reviews that only see the intra-paper details and miss the bigger picture, we’re pretty cooked.
Joseph Suh retweeted
arxiv.org/abs/2608.17981
Recurrent transformer that injects activations from the top layers to the bottom layers at the next step (arxiv.org/abs/2608.08888). Why does this work without training?
Joseph Suh retweeted
Excited to share my first work after joining @MSFTResearch!
LLMs have entered the agentic era, and we now collaborate with agents on complex tasks over many turns of interaction.
But does your agent actually follow what you intended?
We show where agents get lost: LLMs Get Lost in Evolving User Intent.
📄 arxiv.org/abs/2607.20734 🧵 1/N
