Jonibek Mansurov retweeted
Last month we put out a preprint expanding the study of hidden teacher traits that survive knowledge distillation.
Turns out, distillation is quite selective: benchmark knowledge leaks easily, while refusal backdoors don't. 🤨🧵
arxiv.org/pdf/2610.26311 (currently under review)
Jonibek Mansurov retweeted
Finally out 🎉
Introducing HyperBrowseComp: A multilingual, multimodal stress-test benchmark for search agents!
Paper: arxiv.org/abs/2610.03574
Web: hyperbrowsecomp.github.io/
Currently covers 13 languages with handcrafted, 👿diabolical questions.
(See this image as an example)
Jonibek Mansurov retweeted
Happy to announce Ego2Act! A benchmark + eval pipeline for goal-oriented egocentric vidgen
It's fun to see models failing to generate videos of tasks that are trivial for us, but require just a little bit of reasoning or steps...
Congrats @patrickamadeus_ and co-authors!
🤖🤚 Give a video generation model a scene and a goal. Can it generate egocentric execution to get the job done?
We tested 110 real-world scenarios on 6 SoTA models to see how they perform.
Introducing Ego2Act, a goal-directed video generation benchmark 👇
🌐 ego2act.github.io
🧪 alphaxiv.org/abs/2610.01092
🤗 huggingface.co/datasets/ego2…
[🧵Thread]
Jonibek Mansurov retweeted
Like Autorubric, we tackled open classification 2 years before Jev. Our 2024 Statement Tuning paper introduced 125M universal "decision" models matching 200× larger LLMs on unseen tasks. With inference matching Jev’s decision head: arxiv.org/pdf/2404.12897
It gets even deeper...
I spent some time investigating @typesafeai's Jev and striking to see them use a similar abstraction as autorubric -- released 8 months ago and to be presented at @COLM_conf next month. Our criterion types map 1:1 to Jev's Noul/Score/Choice: arxiv.org/abs/2603.00077
Jonibek Mansurov retweeted
So we're building a hard, multilingual multimodal browsing agent benchmark. Data is mostly done, just benchmarking models now. The questions are diabolical😈
Paper & data release coming soon! hyper-browsecomp.github.io
At ICML right now, hit me up if you are interested!
also...
Jonibek Mansurov retweeted
Introducing CultureMix: a joint findings showing that when cultures collide, VLMs collapse!
culturemix.github.io/
3 papers (2 CVPR, 1 NAACL)
30k+ samples
60+ countries & 300+ cultural concepts
up to -58% accu. drop!
#CVPR2026 #NAACL2025 🇺🇸
👇👇👇👇 [1/n]
Jonibek Mansurov retweeted
Replying to @akshay_pachaar
The main (first, last/corresponding) authors are from MBZUAI. The UCL author is also visiting MBZUAI. Please credit them accordingly
nitter.cf/AlhamFikri/status/2045…
Jonibek Mansurov retweeted
Should we treat LLM benchmarking like an annual Olympiad event?🏆
With current benchmarks, it is too easy to overfit tasks or manipulate settings. In some cases, people just cheat / being narrow-tuned to a specific benchmark (*cough* LLaMa-4)
What if we organized an annual, Olympiad-like event? The tasks must be sealed and unknown. Models cannot study for the test. They must be prepared for anything. We explain this in our new position paper.
I am an IOI alum long time ago. I practiced for years to master many algorithms. I wanted to be ready for whatever appeared on the contest day. I believe general LLMs should face the same standard. If they are truly general, they should be ready for whatever use cases.
We propose a flow similar to how we typically organize an Olympiad:
- Call for Task: We propose an open solicitation for challenging, high-quality tasks from the global research community.
- Organizing Committee: A dedicated team curates and improves these submissions. They verify task quality and diversity.
- Model Developers: Developers submit their systems blindly before the tasks are revealed. This prevents teams from iterative gaming or manual tuning once the exam starts.
- The Actual Olympiad: Evaluation happens in a synchronized, short window. The sealed tasks are released, and all models are tested simultaneously to maintain total integrity under the same setting.
Once it is done, everything will be released for reproducibility.
Read the full position paper here: arxiv.org/abs/2603.23292
We worked on this together with my student @jcblaisecruz
Let me know your thoughts!
Jonibek Mansurov retweeted
New position paper!
📄 "LLM Olympiad: Why Model Evaluation Needs a Sealed Exam"
We argue that NLP needs an Olympiad-style event: seal the problems, freeze submissions, run one harness, release everything for audit.
w/ @AlhamFikri
Paper: arxiv.org/abs/2603.23292
Jonibek Mansurov retweeted
As an advisor, my take is that a PhD student is not a paper-generating machine. If a professor thinks AI agents can replace students, they might have lost sight of what advising is truly about
IMO mentorship is about training an independent thinker, building a relationship, and cultivating lifelong bonds. The joy of raising a successful student and watching them flourish is something you simply can't get from prompting an LLM agent
In the last few months, I've spoken to many CS professors who asked me if we even need CS PhD students anymore. Now that we have coding agents, can't professors work directly with agents?
My view is that equipping PhD students with coding agents will allow them to do work that is orders of magnitude more impressive than they otherwise could.
And they can be *accountable* for their outcomes in a way agents can't (yet). For example, who checks the agent's outputs are correct? Who is responsible for mistakes or errors?
Jonibek Mansurov retweeted
Excited to share that we have committed our paper “Vision-Language Models are Confused Tourists” to #CVPR2026 (Findings)! 🇺🇸🏔
Arxiv: arxiv.org/abs/2511.17004
We question whether current SOTA VLMs remain robust in simple cultural grounding QA when distracting contextual objects are present
For example, if you eat chicken schnitzel with Mt. Fuji in the background, will the model fail to recognize it as Japanese katsu?
ConfusedTourists introduces:
👉 5k+ evaluation samples across 3 cultural item categories, comprising 243 unique cultural items from 57 countries and 11 sub-regions 🌍
👉 Evaluation of 14 VLMs across 12 data features 🤖
👉 Findings showing that simple concept mixing can cause up to a -40% drop in perform 📉
Special thanks to my co-authors @IkhlasulHanif0 , @emthehunt, @gentaiscool, @FajriKoto, and my advisor @AlhamFikri for the valuable contributions along the way!
#multimodal #vlm #multicultural #robustness #evaluation #NLProc #ComputerVision
Jonibek Mansurov retweeted
🚀🚨 Sparse-Frontier Major Updates!
You can now evaluate Reasoning + Sparse models at speed, with Sparse-Frontier upgraded to the @vllm_project's v1 engine🔥
We still provide support for Tensor Parallelism and the original sparse attention baselines, but it now works cleanly with newer models, decoding strategies, and evaluation setups. Task coverage and model support were also expanded as part of this release.
The config-based workflow stays the same. If you’re working on sparse decoding, reasoning models, or long-context evaluation, this update makes it easier to run consistent experiments across models, tasks, and attention methods⚡️
Really enjoyed working with @p_nawrot and @PontiEdoardo over the past months to get this release out!
Jonibek Mansurov retweeted
(1/9) Excited to share our new paper🥳,
Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming.
Jonibek Mansurov retweeted
1/11 Proud to share our new paper: SENSIA (SENse-based Symmetric Interlingual Alignment) — a sense-based approach to multilingual adaptation.
Goal: explicit representation-level alignment of meaning.
Jonibek Mansurov retweeted
🎉Silver for @mbzuai at ACPC!!🥈
We also topped the Gulf rankings
Amazing performance by our 1st year UG students in our very first participation
Next: ICPC World Finals 2026🤞
Jonibek Mansurov retweeted
I heard someone canceled their conference trip coz of a threat over this. Juniors could also get their careers hurt for criticizing big name authors
Don't let curiosity get the better of you. Knowing who hates your paper can affect you for years
And please DON'T harass anyone
Jonibek Mansurov retweeted
This is incredible findings – a reproducibility crisis where baselines are not faithfully reproduced or reported (e.g., footnote indicating performance difference)
🍎 In our work (arxiv.org/abs/2505.05408) we tried so hard to ensure apple-to-apple comparison.
Jonibek Mansurov retweeted
Amidst the evaluation/reproducibility crisis for reasoning LLMs, it's great to see *concurrent independent work (with different models & benchmarks) aligns with our findings*!
We reported the same fundamental trade-off: language forcing leads to ✅ compliance, ❌ accuracy!
[1/]💡New Paper
Large reasoning models (LRMs) are strong in English — but how well do they reason in your language?
Our latest work uncovers their limitation and a clear trade-off:
Controlling Thinking Trace Language Comes at the Cost of Accuracy
📄Link: arxiv.org/abs/2505.22888




































