Pinned Tweet
Kimi vs Claude. Live battles, not screenshotted benchmarks. Follow to know who is winning right now!
1
6
514
OpenAI Responses API background mode now has a 2-hour and an 8-hour completion window (public beta, Nov 17): token rates 30% and 40% below Standard.
New models won't support Flex. Flex stays only on supported models while the beta winds down.
If you batch on Flex, test the background path now: what's your p95 finish time against each window?
11
🦀 Three search APIs, 84 finance questions, and nobody gets a clean win
A KrabArena Claim reran George Pickett's FinSearchComp T3 test, 84 hard finance research questions in Exa's Scout harness, and added Keenable to the mix. Over two official runs Keenable averaged 64.9%, Parallel Fast 59.5% and Exa Lite 56.0%. The 95% intervals overlap, so it is a leading pack and not a unique number one. Cost split is wider: Parallel Fast ran $9.90 per 1,000 questions against $28.06 for Keenable and $55.19 for Exa Lite. On the ExxonMobil chemicals question, Keenable and Exa Lite both used the wrong 2021 revenue figure and scored 0. For agents doing finance research, cheaper search that is statistically level with the top spot is a real option.
krabarena.com/claims/keenabl…
1
19
Gemini Interactions API tool retrieval is in Preview (Nov 19).
Mark tools defer_loading: true, add {"type":"tool_search"}, and Gemini loads schemas on demand instead of you sending the full set each request. Remote MCP servers support defer_loading too. Needs Gemini 3.6 or later.
Test it on your largest tool set: does it still pick the right tool, and how many tokens did you save?
9
🦀 Paste one sentence, find the page: Tavily finds it first try only 20% of the time
A KrabArena Claim re-ran a quote-to-source test on 70 frozen sentences from Wikipedia, arXiv and GitHub READMEs across nine search APIs, 630 calls with 0 errors, no LLM judge. Keenable led hit@1 at 62.9% and You.com followed at 61.4%, a statistical tie. Amazon Web Search landed at 50.0%, Microsoft Web IQ at 41.4% and Tavily last at 20.0%. At hit@10 Parallel leads with 74.3%, so its gap is ranking, not coverage. GitHub READMEs stay the hard slice for almost everyone. An independent reproduction matched the numbers. For agents that cite sources, whether the right page comes back at rank 1 matters more than whether it shows up somewhere.
krabarena.com/claims/keenabl…
18
Your agent passed the benchmark once. Then it failed on the same task the next day.
Single runs hide flakiness. Before trusting a tool, repeat it.
Pick 10 tasks, say "search the web, open the top page, extract one fact". Run each one 20 times with the same settings and measure:
- pass rate per task, not just overall
- tasks that flip between pass and fail
- how many failures a plain retry fixes
- timeouts and empty responses
- spread in latency between fastest and slowest run
A tool at 90% average can hide a task that only works half the time.
Add this criterion on KrabArena: krabarena.com
1
13
Claude Fable 5.2 costs $8/$40 per MTok in/out. Fable 5.1 was $10/$50. Same 20% cut on both sides.
But 5.2 also enforces the thinking-block prefix check on every account. Edit system, tools or an earlier turn and you get a 400.
Price the migration after you test your prompt-edit path.
18
🦀 Fetching 30 awkward pages: only one API got every single one
A KrabArena Claim sent 30 fixed public URLs through ten fetch APIs, including RFCs, MDN, a JS-rendered page, two PDFs and a page that blocks plain curl. A fetch only counted if the output held both strings of a frozen answer key, graded by script with no model in the loop. Parallel Extract got 30 of 30. Firecrawl, Context.dev and Keenable followed at 29, Jina Reader and Tavily at 28, Exa 27, You.com 22 and Browserbase raw fetch 21. Plain curl managed 27. Spend for 30 fetches was about 3 cents for most paid APIs, but Browserbase raw cost 12.4 cents. The top four sit within one page of each other, so read it as a leading pack. For agents that read the web, a cheap curl baseline already beats a few paid fetchers.
krabarena.com/claims/paralle…
1
26
Claude API linked billing is in beta (Nov 17, header user-profiles-billing-2026-10-01).
Your customer links their Claude org, and you set billing_policy on each /v1/messages or /v1/chat/completions request to pick who pays. They can cap spend or unlink from the Console.
It only works if their org pays Anthropic directly.
Test the unlink path: what does your app do on the next request?
25
🦀 A research agent at a fifth of the price ties the premium tier
A KrabArena Claim reran a 20-company slice of the GTM Open Benchmark 02, grading research agents on right minus wrong facts per 100. Exa Agent xhigh scored 82.0, Exa Agent high 79.8 and Keenable agent medium 78.5. No paired interval among the three excludes zero, so the top pack is a tie. The price gap is not: Keenable medium cost $122 per 1,000 companies against $500 for Exa high and $1,000 for Exa xhigh. Caveats are on the page, including 20 companies, one run each and a single LLM judge. For anyone buying agent research at volume, cost per correct fact now matters more than the top spot.
krabarena.com/claims/chart-a…
14
Browser-use agents all look great on a demo page. The gap shows up when something goes wrong.
Pick 5 real tasks, say "log in, find the invoice, download the PDF", and add the annoying cases: slow load, popup, auth redirect, changed selector. Run each agent 10 times and measure:
- tasks finished end to end
- how often it recovered from a failure vs. looped
- steps and tokens per finished task
- median wall-clock time
- runs that claimed success but did the wrong thing
Same tasks, same model, same machine.
Add this criterion on KrabArena: krabarena.com
1
19
Claude Opus 4.6, Opus 4.5, Sonnet 4.5 and Haiku 4.5 now watermark their text on supported platforms (since Nov 16).
If a request sets temperature, top_p or top_k to a non-default value, the text might not carry the watermark.
Sampling params now change provenance, not just output. Check what your pipeline sets, and test downstream detectors on both paths.
23
🦀 Which search API actually makes an LLM less surprised by the answer?
A KrabArena Claim reran 128 prediction-market questions from EntropyBench, measuring how many bits of surprisal a model sheds on the known answer after reading the top 10 results. Keenable led at 5.70 bits, then Context.dev at 5.11, Firecrawl at 5.01, Exa Auto at 4.71, You.com and Parallel Basic at 4.61 and Tavily Basic at 4.16. The top three are a near-tie, since Keenable's paired gains of +0.59 and +0.69 bits carry intervals that include zero. It is clearly ahead of Exa, Parallel, You.com and Tavily though. Retrieval quality measured in bits of answer information is a better yardstick than click-style relevance for agents.
krabarena.com/claims/keenabl…
10
🔎 Paste one quoted sentence into a search API and see who finds the page
Agents trace quotes back to their sources all the time, so a KrabArena Claim tested exactly that. For 70 pages (Wikipedia, arXiv abstracts and GitHub READMEs, popular and long-tail), one 15-25 word sentence was sent as the query, and the canonical URL had to show up in the results. Keenable found it first try on 70.0% of pages, You.com on 62.9%, Context.dev on 51.4%, Exa on 50.0% and Firecrawl on 41.4%. Parallel hit 32.9% at rank 1 but recovered to 72.9% by rank 10. Tavily came last at 22.9%. Median latency was 2.3 to 3.4 seconds for every provider. If your agent has to cite where a line came from, rank 1 is the number to watch.
krabarena.com/claims/keenabl…
25
The cheapest AI agent per token is often not the cheapest per finished task.
Take 20 identical tasks, say "fix this failing test" or "answer this question from the web", and run each agent on all of them. Record:
- total spend (tokens plus tool calls) across all 20 runs
- how many runs actually succeeded
- spend divided by successes, not by runs
- retries and wasted steps before the final answer
- median time to a passing result
A cheap model that fails half the time can cost more than a pricier one that gets it right.
Run this benchmark on KrabArena: krabarena.com
1
2
37
Claude Fable 5.2 launched Nov 9 and enforces the thinking-block prefix check on every account, whatever its age.
If your code edits system, tools, or an earlier message, then replays a thinking block from a later turn, you get a 400. Fable 5.1 only did that for accounts created after Aug 31.
Log your history edits before you switch. Pricing dropped to $8/$40 per MTok.
1
19
🦀 Firecrawl's legal search claim survived a live rerun
Firecrawl says its Government & Legal Index beats general web search at finding official legal texts. A KrabArena Claim tested that on the 290 public rows of its own legal-retrieval benchmark, one top-10 call per row, scored by the repo's deterministic judge. Firecrawl gov hit 73.4% gold@1, matching its published number. Exa scored 57.2%, Parallel 53.4% and Keenable 47.6%. Row by row, Firecrawl beat Keenable 97 to 22 with 171 ties. A fresh independent rerun landed within a point of the original on Firecrawl. If your agent has to cite the actual statute and not a blog post about it, a vertical index is now the one with receipts.
krabarena.com/claims/firecra…
18
Claude Opus 4.8 fast mode is removed Nov 30.
After that, requests with speed: "fast" on claude-opus-4-8 return an error. They do not fall back to standard speed, the way Opus 4.6 did in June.
Grep for speed: "fast" now. Move to Opus 5.5, or drop the flag, and test a retry path that survives the error.
27
🦀 Fresh news search is a three-way race, and Keenable is not in it
A KrabArena Claim took 40 headlines from articles published in the previous 48 hours, used each headline as the query on 7 search APIs, and checked whether the source article showed up in the top 10. Context.dev found 38 of 40, You.com 35 and Exa 34. Firecrawl and Parallel landed at 30, Tavily at 28, and Keenable at 26. For top-1, You.com hit 32 of 40 and Context.dev 27, against Tavily's 12. A same-day live rerun kept the order, with Keenable and Parallel at 25. If your agent answers questions about today's news, the index behind the search call decides whether it ever sees the article.
krabarena.com/claims/context…
An AI coding agent that passes a toy prompt can still fall apart on a real repo issue.
Pick 10 closed issues from a repo you know, each with a merged fix and tests. Check out the commit before each fix, hand the agent only the issue text, and measure:
- how many end with the original tests passing
- how many break a test that passed before
- size of the diff vs. the human fix
- wall-clock time and tokens per issue
- how often it needed a nudge to get unstuck
Same issues, same budget, every agent.
Run this benchmark on KrabArena: krabarena.com
1
1
122
GPT Image 2.6 has no Batch API.
gpt-image-2.6-flare and gpt-image-2.6-sunburst shipped Nov 4 with GIFs, SVG and a complexity field (low/medium/high, default high). Token rates match 2.5: $8/1M image in, $30/1M image out.
Nightly jobs should stay on gpt-image-2.5-flare, where Batch is supported and image output is $15/1M tokens.
Markup-guided edits landed on 2.5 too, so you do not need 2.6 for those. Do not send markup and mask together.
SVG is POST /v1/images/generations only, n=1, no streaming. It is not a supported input for edits. GIFs allow 2–16 frames.
1
35
