CS Ph.D @ UIUC Previous: Undergrad @Tsinghua_Uni (21-25)
Joined October 2023
- Tweets33
- Following233
- Followers56
- Likes112
Thanks @HuggingPapers for sharing our work!
Agent memory is paying a tax
Across 11 memory systems, 3 benchmarks and 3 backbones, 90 of 99 memory results land below naive RAG, often at far higher token cost. UIUC and Tsinghua researchers call this gap memory tax.
1
134
Thanks @omarsar0 for sharing our work!
RAG is so back!
Another paper showing that plain RAG keeps winning the agent memory comparison.
They show that many dedicated memory systems score below naive retrieval over the raw history on LongMemEval, PersonaMem, and MINT-Eval, while costing much more.
They split a memory system's value into two parts. Memory benefit counts questions it answers that RAG misses. Memory tax counts questions RAG answers that the memory system gets wrong.
Why does the tax appear?
Building memory rewrites the history. Some facts get dropped or distorted during extraction, and some survive but become harder to retrieve. Memory systems also vary more than RAG when you change the backbone model.
A simple fix helped. Retrieving from the raw history alongside memory and reranking both together reduced the tax.
The practical takeaway here is that before adding a memory layer to your agent, compare it with plain retrieval over the transcript on your own data.
arxiv.org/abs/2610.34073
Chat with Paper: academy.dair.ai/papers/memor…
1
2
422
[1/4] 🤯 Many agent memory systems can’t even beat a naive RAG?
We conducted a comprehensive evaluation and find many underperform the naive RAG, even at higher cost.
To better understand this, we introduce “Memory Tax”, see more in our paper.
arxiv.org/abs/2610.34073
[4/4] Moreover, we find many benchmarks are also retrieval-friendly, the setting makes the memory tax more visible:
When naive RAG already solves many queries, memory has less room for gains and even modest losses become apparent.
2
55
[5/4] The concept of Memory Tax can also be extended to more settings:
- Experimental Memory (experiences, skills, etc)
- World Model Memory.
Besides, we also discuss some effective ways to reduce the tax.
We open-source all the results, see more in:
hexuan21.github.io/MemoryTax…
40
Xuan He retweeted
Building AI tutors requires feedback from students, but real learner studies are slow and hard to scale. 📈
Just in: StudentSim (Microsoft×UIUC) trains personalized AI student simulators 🤖 from real learner 🧑🎓 records and evaluates whether they both match a specific student’s behavior and can respond to tutor guidance.
✏️ Across three learning domains, StudentSim outperforms strong student simulator baselines, including GPT-5.4. In a chess tutor RL study, using StudentSim as guidance helpfulness feedback led to tutor guidance that human experts rated higher in accuracy, guidance quality, and personalization.
📄 Arxiv: 2609.01591
💻 Code: microsoft/StudentSim
Thanks for sharing!
We opensourced all of the:
- bench data (task prompts+start images),
- evaluation results (11 video models and 13 image models), there's also a large gallery in our project page.
Feel free to explore and use them!
HF Data:
huggingface.co/datasets/hexu…
Project Page:
hexuan21.github.io/VGI-Bench…
VGI-Bench
Probing Visual Intelligence in Video Generation Models
paper: huggingface.co/papers/2608.1…
10
843
Xuan He retweeted
As image generation becomes more agentic, benchmarks must evolve with it. AgentGen-Bench tests models on real-world tasks.
@spacexai’s Grok Imagine 2.0 (Low) ranks #2. Great work—we look forward to evaluating the latest model 🚀
huggingface.co/spaces/CongWe…
#AgentGenBench
Xuan He retweeted
How intelligent is your video generation model?
Video generators are emerging as backbones for World Action Models (WAMs). VGI-Bench probes the reasoning and action-relevant priors they encode across 27 tasks and 810 instances.
Thanks for sharing our work!
hexuan21.github.io/VGI-Bench…
1/n Excited to introduce our new work VGI-Bench (EMNLP'2026 Main), a benchmark probing visual intelligence in video generation models: can they solve reasoning tasks by “imagining” visual trajectories?
Project: hexuan21.github.io/VGI-Bench…
HF paper: huggingface.co/papers/2608.1…
HF data: huggingface.co/datasets/hexu…
n/n We hope VGI-Bench becomes a useful testbed for visual reasoning in generative models. The benchmark and data are public, feel free to use it, evaluate your models, build on it, and spread the word!
Paper: arxiv.org/abs/2608.19583
Project: hexuan21.github.io/VGI-Bench…
49
5/n Fruther diagnostics reveal two key findings: appearance matters a lot, changing only the visual style can substantially shift scores and even model rankings. Meanwhile, fine-tuning on large-scale synthetic data yields clear performance gains, showing strong potential for improving visual intelligence through targeted training.
38
4/n We evaluate a broad suite of commercial + open-source video models. Seedance 2.0 leads with only 51.0 overall. Structured Puzzles are hardest; Topology and Temporal are the weakest skills. Current models show emerging visual reasoning, but remain far from reliable.
40
3/n Three gaps motivate VGI-Bench:
① Distribution-mismatched inputs → photorealistic inputs
② Limited visual-rollout demand → process-sensitive tasks requiring trajectories
③ Uncontrolled difficulty/feasibility → pre-screening near the capability boundary + 3 difficulty levels
34
2/n VGI-Bench contains 27 tasks / 810 instances across 4 domains, 7 skill tags, and 3 difficulty levels. Given an image + prompt, models must generate the solution process. We score both goal completion and process validity: Final Score = Completeness × Rubric Score.
136
3/n Three gaps motivate VGI-Bench:
① Distribution-mismatched inputs → photorealistic inputs
② Limited visual-rollout demand → process-sensitive tasks requiring trajectories
③ Uncontrolled difficulty/feasibility → pre-screening near the capability boundary + 3 difficulty levels
4
Updates for the project PlugMem:
🎉Accepted to ICML!
💡Support integration for OpenClaw🦞, claude code🐙 and other framworks!
See more details in the paper
Big PlugMem update 🧠
A plug-and-play memory module for LLM agents — turns raw trajectories into a knowledge graph your agent actually reasons over.
🎉 Accepted to ICML 2026
🔌 Drop it into OpenClaw 🦞, Claude Code, and other agent runtimes
🔍 Visualize memory · test retrieval · replay sessions
🥇 SOTA backbone on LongMemEval & HotpotQA — general enough to build on
Paper: arxiv.org/abs/2603.03296
Code: github.com/TIMAN-group/PlugM…
#ICML2026 #LLM #Agents
Xuan He retweeted
How much of “video understanding” is actually… not about video?
We found that 40–60% of questions in popular benchmarks (VideoMME, MMVU) can be answered without watching the video.
And it gets worse as models scale. 🧵
This problem doesn’t just affect evaluation.
It’s baked into post-training data.
So when you do SFT / RL, a large portion of the “gain” actually comes from better language priors, not better visual grounding.
We propose a simple fix: VidGround
👉 Filter out text-only-answerable questions
👉 Keep only visually grounded data
That’s it.
Surprisingly, less data works better:
• Only 69.1% of the data
• +6.2 pts improvement
• Outperforms more complex RL pipelines
Key takeaway:
- If your data allows shortcutting, your model will learn shortcuts.
- For video understanding:
Grounding signal > data scale > algorithm tricks
📄 huggingface.co/papers/2604.0…
🌐 vidground.etuagi.com/
