PhD at @virginia_tech, supervised by @tuvllms. Work on LLMs' long-context search, post-training, and self evolution. ex-Research Intern at @AdobeResearch.
Blacksburg, VA
Joined May 2022
- Tweets256
- Following401
- Followers141
- Likes619
Pinned Tweet
🚀 I am excited to share our recent work, which I will present at #COLM2026
🚨 Our data curation framework improves LLMs’ long-context reasoning capability in open domains, even via self-distillation!
We introduce Pi^2, a pipeline that 1) transforms Wikipedia tables into high-quality long-context QA data and 2) back-translates reasoning traces with realistic context.
📜: openreview.net/forum?id=4yDp…
💻: github.com/vt-pi-squared/pi-…
🧵:👇
(1/n)
What a fascinating emergent capability of frontier models!
It would definitely be helpful for recursive self-improvement (RSI).
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Quyet V. Do retweeted
i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.
Quyet V. Do retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Quyet V. Do retweeted
I recommended desk rejections (DR) for 5 out of 14 submissions, as an #ICLR2027 Area Chair.
Story behind this: I did not use any LLM detection tools, which are unnecessary and should not be used for judging DR. These submissions shared some shockingly basic problems in scientific writing, and therefore each paper only took me about 10 mins (thank God) to identify and report the issues.
1. Undefined jargon. Writing 101: Your paper must be self-contained. If a term isn't standard knowledge in the field, it should be defined.
2. Not analyzing or pretending to analyze results. Experimental analysis is about providing insights, not simply repeating numbers from figures and tables. Bro, I can read the numbers myself.
3. Missing citations. Scientific claims need support, either from existing literature or in-paper empirical evidence. Fewer than three citations in an entire introduction? Either the work is a once-in-a-century masterpiece, or the claims are hallucinated in the paper.
My biggest concern isn't whether these papers were written using LLMs. It's that some submissions fail to meet the most basic standards of scientific communication and therefore should not be reviewed.
Considering this has already wasted some reviewers' time, I strongly recommend introducing AI to flag such papers at an earlier stage.
Thanks for ICLR's policy on desk rejections. Hope this can save some time for the responsible reviewers.
Quyet V. Do retweeted
Small RL companies are the AI industry's equivalent of quant shops. Let me explain.
Both are small, secretive, full of brilliant people, profitable, and always one step from their edge disappearing.
The parallels:
- Secretive. Nobody publishes their signals. Labs won't say what they're training next, and vendors can't say who they sell to. Epoch calls the market "largely behind closed doors."
- Hunting alpha. For a quant, a trade stops working once everyone finds it. For these shops, a task stops paying once models can solve it (labs drop tasks at around 70% pass rate). Meanwhile, labs are building their own in-house. So they always have to find the next field.
- Deep Industry Connections. You need to know the problem cold to get in. Founders have usually worked at, or sold to, the labs. You can't define "correct" for fund accounting without having done fund accounting.
- They buy strange data. Quants buy satellite photos of parking lots. RL shops bid for enterprise data (eg $12.5M bid for Spirit Airlines historical data).
- Brilliant people. Quants hire math olympiad winners. These shops hire ex-lab researchers and real practitioners.
- Huge revenue per head. Jane Street made about $11M per employee in 2025. Proximal is about $5M. The tweet's company would be $20M.
- One narrow problem that makes something more efficient. Quants make markets more efficient. These RL shops make models training more efficient.
- High risk (for different reasons). A quant can lose millions in a day on a bad trade. An RL shop's risk is that one lab trains a model that makes their environments irrelevant, or brings the work in-house, and the revenue stops.
- Overfitting/reward hacking. A quant's nightmare is a strategy that looks perfect on historical data and fails live. An RL shop's nightmare is an RL grader that scores perfectly and trains the model incorrectly.
- Infrastructure arms race. Quants pay for microseconds of speed. RL shops race to spin up thousands of disposable computers per training run.
Obviously there are as many differences between RL companies and quant shops, but right now the aura around them feels eerily similar.
Curious to see how this market matures as robotics opens up a new surface, and whether the frontier labs end up absorbing them.
Quyet V. Do retweeted
PhD Advising vs Vibe Coding #1:
One of the key skills a PhD advisor learns is context switching. Students come into your office every 30-60 minutes and you need to remember their project, where you left off, and what they were supposed to do next. It's a learned skill and exactly what you need to manage AI agents.
I’ll be presenting my #COLM2026 paper today 👋
📜 Pi^2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
📍Location: Poster Session 1 (11am-1pm), #134, Grand Ballroom.
🌐︎ Virtual site: colm.eventhosts.cc/virtual/2…
If you are interested in this paper, I would love to have you visit the poster and discuss the implications of our findings.
☕️ I am also eager to chat about other topics, such as
- AI self-evolution
- search over long-context
DM me if you're around!
See you there!
Quyet V. Do retweeted
also, my awesome PhD student @Quyet_Azir is presenting his work on curating long-context reasoning data today (Oct 6) 11 AM-1 PM PT @ Grand Ballroom Poster #134 Poster Session 1.
nitter.cf/Quyet_Azir/status/2106…
🚀 I am excited to share our recent work, which I will present at #COLM2026
🚨 Our data curation framework improves LLMs’ long-context reasoning capability in open domains, even via self-distillation!
We introduce Pi^2, a pipeline that 1) transforms Wikipedia tables into high-quality long-context QA data and 2) back-translates reasoning traces with realistic context.
📜: openreview.net/forum?id=4yDp…
💻: github.com/vt-pi-squared/pi-…
🧵:👇
(1/n)
Quyet V. Do retweeted
I'm not at @COLM_conf, but my awesome PhD student @noahpro99 is giving a talk on our EvoSkill work today at the COLM Social: Self-Improving Agents (Oct 6 1-2:30 PM PT, Franciscan D). Come chat with him about his first-hand experience with different skill evolution frameworks!
hosting the Self-Improving Agents social at @COLM_conf tomorrow
- 4 talks: @sethkarten @yoonholeee @qinyuan_ye and Noah Provenzano
- panel: @HuaxiuYaoML @ZhiruoW @jasonwu0731
my 4th one after ICLR, ICML and NeurIPS, and I still love hosting these.
1-2:30pm, Franciscan D. come by if you're at COLM
luma.com/m8v4fq76
[Ablation Studies]
💡 Other findings:
1. Self-distillation works: gpt-oss-20b improves +4.4% using its own reasoning traces (competitive with using a larger open-source teacher model)
2. Data efficiency: Just SFT-ing with 100 high-quality samples yield +1.34 to +2.85 gain, validating our curation quality
(8/n)
🤝 Work done w/ my wonderful collaborators and advisor:
@thinhphp_vt, @NguyenMusaic, Sha Li, Pratibha Zunjare, and @tuvllms
(n/n)
Appreciate your attention 🤗
Questions? Drop them below! 🧵
Quyet V. Do retweeted
Glad to share that our paper on The Master Key Hypothesis / cross-model capability transfer got accepted @NeurIPSConf 2026 as spotlight 🎉 !! Top 1.3% (292 spotlight + 112 oral) out of 30709 submissions. Congrats to my students @Sub_RBala @linusdd44804 @helix9550 and co-authors @mohitban47!!
🌟New paper alert: The Master Key Hypothesis🌟
Post-training is expensive AND you have to repeat it for every new model. What if you could just transfer the capability instead?
We introduce UNLOCK — a training-free and label-free framework that transfers capabilities across models using low-rank linear projections.
Takeaways:
1⃣ Skills like CoT & math reasoning live as directions in the model's latent space. We capture these as a steering vector — we call this the Master Key.
2⃣ Simple low-rank linear transformations are sufficient to transfer the Master Key across models' latent spaces
3⃣ UNLOCK elicits behaviors that even prompting can't reliably trigger
4⃣ Gains scale with base model strength. Stronger models show larger gains post-transfer.
5⃣ UNLOCK works by sharpening the output distribution — steering the model toward the right answer.
