Klu.ai (@klu_ai) header
Klu.ai (@klu_ai) profile picture

Ship AI Today / Core team @0xSMW @SSabev @brunotorious @abielzulio

San Francisco, CA
Joined February 2023
Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)Photo by Klu.ai (@klu_ai)
Refreshed LLM Leaderboard with the hardest benchmarks for today's leading models. klu.ai/llm-leaderboard
Refreshed the @klu_ai LLM leaderboard with a new section for the hardest benchmarks on multi-modal models, including GAIA scores (no tools, scratchpad for thinking). klu.ai/llm-leaderboard
2
1,634
time to retire MMLU, HellaSWAG, HumanEval, BBHard, GSM8K, and many other benchmarks. leading models rank in/near 90th percentile. GPQA, MMMU, BigCodeBench, GAIA, QUAKE, and others offer better measures for 2024 and beyond. klu.ai/llm-leaderboard
1
2
5
925
QUAKE: A new benchmark reveals frontier AI models score only 28% on practical tasks, despite 80%+ on standard evals – here's why it matters...
We built a better model eval benchmark... Introducing QUAKE: multi-modal use case eval across 8 domains and 9 task categories performed by today's knowledge workers. We found that frontier models score an average of 28% compared to the saturated +80% on MMLU and others.
1
2
397
Finetuning and evaluation made possible with Klu.ai platform. The fastest way to optimize your LLM-powered features.
Reached above 50% on ARC AGI. Spent the morning testing a few ideas stuck in my mind. GPT-4 Turbo is ~45% better than GPT-4o. Built a few-shot dataset from where GPT-4 Turbo outperforms. Tested system message improvement, threaded n-shots, and a GPT-4o fine-tune.
1
4
422
Mistral Large on La Plateforme is fast... 25% faster than Claude 2... Same TPS as Gemini Pro, which itself is 30% faster than gpt-3.5-turbo... more soon...
Excited to test out further, but early prompts are looking quite nice. Very strong logic when debugging typescript with multiple dependencies.
2
1
995
"We become what we behold. We shape our tools and then our tools shape us."
This Post is from an account that no longer exists. Learn more
1
472
Excited to test out further, but early prompts are looking quite nice. Very strong logic when debugging typescript with multiple dependencies.
We’re announcing a new optimised model today! Mistral Large has top-tier reasoning capacities, is multi-lingual by design, has native function calling capacities and a 32k model. The pre-trained model has 81.2% accuracy on MMLU. Learn more on mistral.ai/news/mistral-larg…. Mistral Large is available on la Plateforme, as well as on Azure, following our commitment to bring frontier AI everywhere. We're eager to see what developers do with Mistral Large!
1
1,273
This week Gemini Pro came out for developers, but was largely overshadowed by Google's demo controversy and the release of Mixtral. Overall, Gemini Pro is a good alternative to gpt-3.5-turbo – let me show you what we found. klu.ai/blog/gemini-pro-vs-gp…
1
1
2
478
Gemini Pro is better at reading comprehension and instruction following than gpt-3.5-turbo in real-world scenarios. It's a good model, just a bit tarnished by the Google vibes. (PS. Correct answer here is D)
1
325
Oh – one more thing – it is extremely locked down, refusing, among other things, to generate code for the OpenAI SDK.
157
History will write that the token wars began in 2024, but we'll know they really started in December.
$0.27/1M tokens, lol this is cheapest I’ve seen Mixtral being served and it dropped significantly in price in just a week
1
264
You had me at Supafly.
LW🆇 Day 5 - Build Stage 🏗️ announcement Fly Postgres, a managed Postgres offering jointly developed by Supabase and @flydotio. Learn more: supabase.com/blog/postgres-o…
1
1
160
Shout out to Google Palm 2 Otter and Unicorn. RIP buddies, we hardly ever knew you.
95