Jonibek Mansurov (@M_Jonibek) header
Jonibek Mansurov (@M_Jonibek) profile picture
Joined May 2023
Last month we put out a preprint expanding the study of hidden teacher traits that survive knowledge distillation. Turns out, distillation is quite selective: benchmark knowledge leaks easily, while refusal backdoors don't. 🤨🧵 arxiv.org/pdf/2610.26311 (currently under review)
2
6
15
473
Jonibek Mansurov retweeted
Finally out 🎉 Introducing HyperBrowseComp: A multilingual, multimodal stress-test benchmark for search agents! Paper: arxiv.org/abs/2610.03574 Web: hyperbrowsecomp.github.io/ Currently covers 13 languages with handcrafted, 👿diabolical questions. (See this image as an example)
15
20
5
87
4,676
Jonibek Mansurov retweeted
Happy to announce Ego2Act! A benchmark + eval pipeline for goal-oriented egocentric vidgen It's fun to see models failing to generate videos of tasks that are trivial for us, but require just a little bit of reasoning or steps... Congrats @patrickamadeus_ and co-authors!
🤖🤚 Give a video generation model a scene and a goal. Can it generate egocentric execution to get the job done? We tested 110 real-world scenarios on 6 SoTA models to see how they perform. Introducing Ego2Act, a goal-directed video generation benchmark 👇 🌐 ego2act.github.io 🧪 alphaxiv.org/abs/2610.01092 🤗 huggingface.co/datasets/ego2… [🧵Thread]
5
34
2,567
Jonibek Mansurov retweeted
Like Autorubric, we tackled open classification 2 years before Jev. Our 2024 Statement Tuning paper introduced 125M universal "decision" models matching 200× larger LLMs on unseen tasks. With inference matching Jev’s decision head: arxiv.org/pdf/2404.12897 It gets even deeper...
I spent some time investigating @typesafeai's Jev and striking to see them use a similar abstraction as autorubric -- released 8 months ago and to be presented at @COLM_conf next month. Our criterion types map 1:1 to Jev's Noul/Score/Choice: arxiv.org/abs/2603.00077
1
3
2
10
3,231
Jonibek Mansurov retweeted
So we're building a hard, multilingual multimodal browsing agent benchmark. Data is mostly done, just benchmarking models now. The questions are diabolical😈 Paper & data release coming soon! hyper-browsecomp.github.io At ICML right now, hit me up if you are interested! also...
Soo... benchmarking work can cost a non-trivial amount of money too More on this work soon If anyone wants sponsor some API credits to see if their models can actually survive, lmk🫠
3
14
2
50
5,140
Jonibek Mansurov retweeted
Introducing CultureMix: a joint findings showing that when cultures collide, VLMs collapse! culturemix.github.io/ 3 papers (2 CVPR, 1 NAACL) 30k+ samples 60+ countries & 300+ cultural concepts up to -58% accu. drop! #CVPR2026 #NAACL2025 🇺🇸 👇👇👇👇 [1/n]
1
8
2
15
2,180
Jonibek Mansurov retweeted
Replying to @akshay_pachaar
The main (first, last/corresponding) authors are from MBZUAI. The UCL author is also visiting MBZUAI. Please credit them accordingly nitter.cf/AlhamFikri/status/2045…
"Researchers from UCL" but they are all from MBZUAI, even the first and corresponding authors Give proper credit please. You don't have to go this low for clout
4
53
3,403
Jonibek Mansurov retweeted
Should we treat LLM benchmarking like an annual Olympiad event?🏆 With current benchmarks, it is too easy to overfit tasks or manipulate settings. In some cases, people just cheat / being narrow-tuned to a specific benchmark (*cough* LLaMa-4) What if we organized an annual, Olympiad-like event? The tasks must be sealed and unknown. Models cannot study for the test. They must be prepared for anything. We explain this in our new position paper. I am an IOI alum long time ago. I practiced for years to master many algorithms. I wanted to be ready for whatever appeared on the contest day. I believe general LLMs should face the same standard. If they are truly general, they should be ready for whatever use cases. We propose a flow similar to how we typically organize an Olympiad: - Call for Task: We propose an open solicitation for challenging, high-quality tasks from the global research community. - Organizing Committee: A dedicated team curates and improves these submissions. They verify task quality and diversity. - Model Developers: Developers submit their systems blindly before the tasks are revealed. This prevents teams from iterative gaming or manual tuning once the exam starts. - The Actual Olympiad: Evaluation happens in a synchronized, short window. The sealed tasks are released, and all models are tested simultaneously to maintain total integrity under the same setting. Once it is done, everything will be released for reproducibility. Read the full position paper here: arxiv.org/abs/2603.23292 We worked on this together with my student @jcblaisecruz Let me know your thoughts!
4
15
2
74
7,071
Jonibek Mansurov retweeted
New position paper! 📄 "LLM Olympiad: Why Model Evaluation Needs a Sealed Exam" We argue that NLP needs an Olympiad-style event: seal the problems, freeze submissions, run one harness, release everything for audit. w/ @AlhamFikri Paper: arxiv.org/abs/2603.23292
4
8
1
28
2,783
Jonibek Mansurov retweeted
As an advisor, my take is that a PhD student is not a paper-generating machine. If a professor thinks AI agents can replace students, they might have lost sight of what advising is truly about IMO mentorship is about training an independent thinker, building a relationship, and cultivating lifelong bonds. The joy of raising a successful student and watching them flourish is something you simply can't get from prompting an LLM agent
In the last few months, I've spoken to many CS professors who asked me if we even need CS PhD students anymore. Now that we have coding agents, can't professors work directly with agents? My view is that equipping PhD students with coding agents will allow them to do work that is orders of magnitude more impressive than they otherwise could. And they can be *accountable* for their outcomes in a way agents can't (yet). For example, who checks the agent's outputs are correct? Who is responsible for mistakes or errors?
6
31
3
238
23,128
Jonibek Mansurov retweeted
Excited to share that we have committed our paper “Vision-Language Models are Confused Tourists” to #CVPR2026 (Findings)! 🇺🇸🏔 Arxiv: arxiv.org/abs/2511.17004 We question whether current SOTA VLMs remain robust in simple cultural grounding QA when distracting contextual objects are present For example, if you eat chicken schnitzel with Mt. Fuji in the background, will the model fail to recognize it as Japanese katsu? ConfusedTourists introduces: 👉 5k+ evaluation samples across 3 cultural item categories, comprising 243 unique cultural items from 57 countries and 11 sub-regions 🌍 👉 Evaluation of 14 VLMs across 12 data features 🤖 👉 Findings showing that simple concept mixing can cause up to a -40% drop in perform 📉 Special thanks to my co-authors @IkhlasulHanif0 , @emthehunt, @gentaiscool, @FajriKoto, and my advisor @AlhamFikri for the valuable contributions along the way! #multimodal #vlm #multicultural #robustness #evaluation #NLProc #ComputerVision
5
14
3
44
10,976
Jonibek Mansurov retweeted
🚀🚨 Sparse-Frontier Major Updates! You can now evaluate Reasoning + Sparse models at speed, with Sparse-Frontier upgraded to the @vllm_project's v1 engine🔥 We still provide support for Tensor Parallelism and the original sparse attention baselines, but it now works cleanly with newer models, decoding strategies, and evaluation setups. Task coverage and model support were also expanded as part of this release. The config-based workflow stays the same. If you’re working on sparse decoding, reasoning models, or long-context evaluation, this update makes it easier to run consistent experiments across models, tasks, and attention methods⚡️ Really enjoyed working with @p_nawrot and @PontiEdoardo over the past months to get this release out!
1
6
3
37
18,564
Jonibek Mansurov retweeted
(1/9) Excited to share our new paper🥳, Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming.
2
8
5
22
5,746
Jonibek Mansurov retweeted
1/11 Proud to share our new paper: SENSIA (SENse-based Symmetric Interlingual Alignment) — a sense-based approach to multilingual adaptation. Goal: explicit representation-level alignment of meaning.
2
7
3
24
3,599
Jonibek Mansurov retweeted
Most culture test benchmark is mostly static, which may lead to data saturation and leakage, hence making the score not reliable to measure the capability of LLMs. Thus, we benchmark these LLMs to play a social deduction game!
1
9
2
21
3,330
Jonibek Mansurov retweeted
🎉Silver for @mbzuai at ACPC!!🥈 We also topped the Gulf rankings Amazing performance by our 1st year UG students in our very first participation Next: ICPC World Finals 2026🤞
3
9
4
60
14,660
Jonibek Mansurov retweeted
I heard someone canceled their conference trip coz of a threat over this. Juniors could also get their careers hurt for criticizing big name authors Don't let curiosity get the better of you. Knowing who hates your paper can affect you for years And please DON'T harass anyone
3
1
42
4,585
Jonibek Mansurov retweeted
This is incredible findings – a reproducibility crisis where baselines are not faithfully reproduced or reported (e.g., footnote indicating performance difference) 🍎 In our work (arxiv.org/abs/2505.05408) we tried so hard to ensure apple-to-apple comparison.
Confused about recent LLM RL results where models improve without any ground-truth signal? We were too. Until we looked at the reported numbers of the Pre-RL models and realized they were serverely underreported across papers. We compiled discrepancies in a blog below🧵👇
3
8
899
Jonibek Mansurov retweeted
Amidst the evaluation/reproducibility crisis for reasoning LLMs, it's great to see *concurrent independent work (with different models & benchmarks) aligns with our findings*! We reported the same fundamental trade-off: language forcing leads to ✅ compliance, ❌ accuracy!
[1/]💡New Paper Large reasoning models (LRMs) are strong in English — but how well do they reason in your language? Our latest work uncovers their limitation and a clear trade-off: Controlling Thinking Trace Language Comes at the Cost of Accuracy 📄Link: arxiv.org/abs/2505.22888
11
20
3,421
Jonibek Mansurov retweeted
Can an AI trained in English solve math problems in other languages without extra training?
17
72
3
586
35,790