Mari Canwiper retweeted
Edit the photo. Keep the person.
Ideogram 4.5 ranks #1 among 20 models for preserving facial identity after five consecutive edits in @RapidataAI’s benchmark.
After 5 edits, does the person still look the same? Ideogram 4.5 ranks #1 among 20 models for preserving facial identity! 🥇
We benchmarked character identity drift across 199 photos, applying 5 consecutive edits and evaluating only the resulting faces. This benchmark holds 322K+ human judgments.
All outputs, judgments and methodology are publicly available on Benchmark AI.
They finally started caring about SVG performance on the Fable model line:
Fable 5: 1112 elo
Fable 5.1: 1164 elo
Fable 5.2: 1489 elo (!)
Interestingly, Opus 5.5 (1357 elo) had received the SVG treatment a while ago already.
Fable 5.2 enters our SVG benchmark at the #1 position.
The benchmark contains 4M human judgments across prompt alignment, coherence, and preference.
All outputs, votes, and methodology are public on benchmark.ai/svg.
1
66
Congrats on being best in camera movement!
Moving around in a world is only the beginning though.
Next step: nailing physics simulations
Next next step: accurately predicting behaviour of humans and other entities
Human benchmarks will remain the ultimate eval throughout.
Atlas ranks #1 in @RapidataAI's camera-movement benchmark, which evaluates how well models follow camera instructions.
The evaluation compares 19 world and video models across 600 prompts, with 513K+ human judgments.
More on the results and methodology below.
2
133
Soon we’ll be at half a million!
2
65
It was a pleasure to talk about human evals last Monday and share a preview of one of our upcoming LLM benchmarks! Especially since I was sharing the stage with my two friends @flschottmann and @MikeDoesDo.
Creativity and diversity go hand in hand. While post-training improves most benchmark scores, it decreases creativity. Very excited to see a genuinely fresh angle in the gen-ai benchmaxxing landscape!
The best models are the worst at diversity: the better models get at prompt alignment, the less diverse their outputs tend to be (r = −0.77).
Qwen and SeeDream seem to be different in their post-training method, being able to maintain diversity.
Compare outputs and explore the study at benchmark.ai/output-diversit…
3
60
Mari Canwiper retweeted
Following the trend, here are 780K+ human votes on how well LLMs can generate 3D objects from text using code. Our team had so much fun scrolling through the outputs.
Gemini 4 Argon takes #1, followed by GPT-6 Astra.
Explore all outputs, votes and rankings on benchmark.ai
Mari Canwiper retweeted
"AMT done right!"
@Taesung & @donghoon_ahn used 1600 human judgements to eval quality and diversity in their new paper "Correction Training for Diffusion Models"
Thanks for the kind words!
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
🙏
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
1
25
Mari Canwiper retweeted
BREAKING: Nano Banana 2.1 just hit the shelves and is the now the best non-OpenAI available (Reve was bought)!
Results are backed by over 64'000 ratings from humans of all walks of life on just this model and all 6.5M ratings are free to download and use. Maximum transparency!
Mari Canwiper retweeted
Releasing 6M+ human preferences for T2I models. 43 models, 1,514 prompts, 680K+ pairwise comparisons, with separate judgments for prompt alignment, visual preference, coherence, and text rendering.
Other datasets from Benchmark.AI can also be found there ;). Comment if there is a dataset that you'd like us to work on for your research!
huggingface.co/datasets/Rapi…
🤖 Made with AI
Mari Canwiper retweeted
BREAKING: @bfl_ai FLUX 3 Image enters the Pareto Frontier in the first public Benchmark of the model!
With over 64k annotations collected in the hours since launch the results show once again that Black Forest Labs plays in the top league, competing with the trillion dollar hyperscalers!
The model was evaluate across 4 categories.
Alignment, Coherence, Preference and Text-Coherence
Full Results here: benchmark.ai/image
All data for download at: huggingface.co/datasets/Rapi…
Accurate camera control is crucial for filmmaking.
According to 900K+ human judgments, Gemini Omni Fash & Wan 3.0 take the top spots in our World Models Camera Motion benchmark.
Starting from a static image, we ask models to generate a video following a specific camera movement, then ask humans which output follows the instruction best.
Congrats to Google DeepMind and Alibaba Cloud.
All outputs + matchups are on Benchmark AI. Full data downloadable on HF.
39
Mari Canwiper retweeted
According to 900K+ human judgments, Gemini Omni Fash & Wan 3.0 take the top spots in our World Models Camera Motion benchmark.
Starting from a static image, we ask models to generate a video following a specific camera movement, then ask humans which output follows the instruction best.
Congrats to Google DeepMind and Alibaba Cloud.
All outputs + matchups are on Benchmark AI. Full data downloadable on HF.
Mari Canwiper retweeted
Visual realism ≠ physics. Based on Physics-IQ, a @GoogleDeepMind paper, we benchmarked 26 world models on 65 real-world physics scenarios, 450K+ human judgements
Gemini Omni 1.1 Flash takes #1, followed by Minimax H3.
Outputs & votes are visible on benchmark.ai
Mari Canwiper retweeted
We asked random people around to world to draw letters, giving us the most international font possible. Each letter is from a different person from a different country, making it the most representative font possible. @UN should adopt it immediately.
Mari Canwiper retweeted
Millions are spent training models for specific capabilities, then judging them with: “Which output do you prefer?” 💸
We need more granularity + transparency. So here comes benchmark.ai.
First benchmark: SVG generation. 42 models, 1.9M+ human judgements.
Prompts, outputs, match-ups & methodology are public. What benchmark do you want to see next?
A Rapid is a single question, answered in less than 30 seconds, inside a mobile app. Millions a day. Good design means good data. Only very few people can pull this off, if you're one of them:
rapidata.ai/join/uxui-produc…
1
2
44












