6K+ human judgements/minute via API for Online RLHF and Benchmarks. Advancing SOTA GenAI research through weekly, open benchmarks → https://nitter.cf/t.co/xJHzy9dKbE
Joined June 2024
- Tweets162
- Following185
- Followers142
- Likes542
Watching people we curated from across the globe to help with evaluating or training models is one of the most satisfying things one can do.
Building on top of:
Flow-GRPO (@jie_liu1 et al.), but with real humans instead of a reward model.
DPO (@bram_wallace et al.), but online instead of offline.
How it works: rapidata.ai/blog/live-human-…
In academia? Our research grants give up to $50K in annotation credits to try it.
Kimi K3.1 Pro takes #1 among open-weight models on and #7 overall among 71 models on our SVG benchmark. An open-weight model outperforming some of the best closed-weight models is very rare to see 👀.
Full rankings, and all data available on benchmark.ai.
Targeted post-training is increasingly what makes models excel at specific workflows. Great to see @PrunaAI pushing this forward! Congrats team!
P-Video-2-Pro-Avatar is #1 in overall human ratings, at a fraction of the cost and in seconds.
- Quality: #1 of 10 models at 768p
- Cost: 7× cheaper, just $0.02/s vs $0.14/s at 480p
- Speed: 3× faster, just ~9.7s for a 10s clip at 480p
- # 1 for music videos at 768p and #2 at 480p
Across both Quality × Price and Quality × Speed, P-Video-2-Pro-Avatar stands out in the high-quality, low-cost, fast-generation region.
From UGC ads to music videos, create high-quality talking avatars without compromising on speed or budget.
Benchmarked by @RapidataAI.
🧩 API & Playground: buff.ly/OHtx3eM
📚 Documentation: buff.ly/bHQHKV0
📊 P-Bench: buff.ly/Fk54Z2K
Pairwise preferences from all our benchmarks can be downloaded on: huggingface.co/Rapidata
Fibo 2.0 ranks #2 amongst open-weight models in our T2I benchmark including 7M+ human judgements (700K on open-weight matchups).
1.5K prompts, all outputs, matchups, and votes can be inspected on Benchmark.ai
Following the trend, here are 780K+ human votes on how well LLMs can generate 3D objects from text using code. Our team had so much fun scrolling through the outputs.
Gemini 4 Argon takes #1, followed by GPT-6 Astra.
Explore all outputs, votes and rankings on benchmark.ai
As Benchmark AI has more World Model benchmarks, we used the FLUX 3 Max launch to compare three models across 4 benchmarks, based on 3M+ human judgments.
Our goal with Benchmark AI is to better understand which models are best at what. This is a first step :)
PS: all outputs, matchups, votes, prompts & methodology are transparent and inspectable
Clever approach to diffusion denoising, improving both image quality and diversity. The use of Rapidata in this paper is is exactly why we launched our research grants.
Can diffusion models generate high-quality images without CFG?
Yes! We introduce ✨Correction Training for Diffusion Models✨, which significantly improves sample quality without relying on classifier-free guidance.
Project page: sunovivid.github.io/correcti…
🧵 (1/n)
Affordable, fast, and this time also with high control. Benchmark will be shared soon!
Introducing P-Video-2-Pro-Reference based on @MiniMax_AI H3! 🎬
Combine images, videos, and audio references to guide how your video looks, moves, and sounds. Keep characters consistent, transfer motion, and bring different visual elements together in a single generation.
• 𝗜𝗻𝗽𝘂𝘁: Text prompt with image and/or video references, plus optional audio
• 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀: Up to 9 images, 3 videos, and 3 audio files
• 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻: 5–15s at 480p or 768p
• 𝗣𝗿𝗶𝗰𝗶𝗻𝗴: Output starting at $0.015/s in Cost mode, with reference inputs billed separately
• 𝗖𝗼𝗻𝘁𝗿𝗼𝗹: First and last frames, seed, aspect ratio, and prompt upsampling
Two modes: Speed and Cost. Choose the trade-off that fits your workflow.
Available via our inference partners @Cloudflare, @scenario_gg, @replicate, @WiroAI, @TellersAI, @eachlabs, @wavespeed_ai, @inference_sh, @_segmind, @togethercompute, @EdenAIco, and @runware
Validated by our benchmark partners @datapointai, @RapidataAI, and @DesignArena
📚 Model docs: docs.api.pruna.ai/guides/mod…
🧩 API: dashboard.pruna.ai/login
Rapidata retweeted
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
"AMT done right!"
@Taesung & @donghoon_ahn used 1600 human judgements to eval quality and diversity in their new paper "Correction Training for Diffusion Models"
Thanks for the kind words!
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
New Benchmark: Image-to-3D with 307K+ human votes across 14 models.
Hi3D v3.0 leads the leaderboard, with Meshy v7.1 and Tripo H3.1 among the top ones.
We evaluated geometry and visual faithfulness across 357 input images.
Full results on benchmark.ai
World Models Realism: We benchmarked world models with camera footage. Giving models a real starting frame, we asked them to predict what happens next. Then 500K+ humans compared the outputs against each other (including the real camera footage).
Camera footage still ranks number 1 by far (still expected tbf haha). But on the models' side:
@grok Image Video 1.5 is #1 for Photorealism,
@MiniMax_AI H3 is #1 for Visual Integrity.
@bfl_ai Flux 3.0 and Gemini Omni Flash 1.1 are following just behind it :).
In some examples, telling generated from real is genuinely hard. Check out the full leaderboard, and explore all match-ups and outputs on: benchmark.ai/world-models-re…