Towards AI

The easiest way to know you've actually learned LLM inference: Hit one measurable number at each stage before you move on. Here's the 90-day plan we'd follow, with the target for each stage: - Days 1–20, the math: reach 60% of your GPU's decode ceiling at batch 1 (Databricks measured 60% on H100s) - Days 21–45, serving engines: 2x the throughput of static batching at the same latency - Days 46–70, optimization: 2x lower per-token latency at batch 1, keeping 99% of full-precision accuracy - Days 71–90, production: serve Llama 3.1 8B at MLPerf's interactive limits (p99 first token ≤ 500 ms, ≤ 30 ms per output token) Why numbers? Because a topic list can't tell you when you've learned something. A target can. Here's the math behind stage 1, for Llama 3.1 8B in BF16 on one H100: ↳ every generated token re-reads all 16 GB of weights ↳ 3.35 TB/s of memory bandwidth ÷ 16 GB ≈ 209 tok/s, the batch-1 ceiling ↳ the KV cache adds 2 × 32 layers × 8 KV heads × 128 dims × 2 bytes = 128 KB per token ↳ one full 128K-token request needs 17 GB of KV cache, more than the weights So decode speed is set by memory bandwidth. That same math explains the next two stages. Batching works because an H100 does about 295 FLOPs per byte it reads (989 TFLOPS ÷ 3.35 TB/s), and decode uses about 1 per request. Extra requests ride on the same weight reads until the batch is in the hundreds. Speculative decoding fades at scale for the same reason. EAGLE-3 reports up to 6.5x faster decoding, but 1.38x more throughput at batch 64. The graphic below maps all 90 days, with the best resource for each stage. Start with the inference chapter of How to Scale Your Model: jax-ml.github.io/scaling-boo… (bookmark it)

Cloudflare just admitted its AI leaked customer data! (what broke in the KV cache, step by step, plus 5 checks so it doesn't happen to you) On Workers AI, 155 responses from the GLM 5.3 model returned text from earlier requests sent by other customers. Cloudflare took the model offline within 40 minutes of declaring the incident, fixed the setup the same day, and shared the full technical root cause openly. The interesting part is why it happened. In SGLang, the serving engine used here, prefill and decode disaggregation splits prompt processing and token generation across separate GPUs. During prefill, the model builds the KV cache, the stored attention vectors that let later tokens read earlier context. ↳ SGLang sizes the KV cache pool from leftover GPU memory at startup, which came out to 681,280 tokens, smaller than the 1,048,576-token advertised context window. ↳ When a prompt exceeded that pool, the engine flagged it as rejected and replaced the prompt with a one-token placeholder. ↳ The check that guards the handoff only looked at whether the request had finished, so it still looked live and got sent to the decode node instead of returning an error. ↳ KV cache pages are not wiped between requests, so the decode node returned a coherent HTTP 200 reply while reading stale pages left by earlier requests. The diagram below depicts the complete flow. Here is our checklist for teams self-hosting LLMs: - Assert at startup that every node's KV pool covers the advertised context, or fail the deploy. - Send an over-limit prompt in staging and expect an HTTP 400 error, not a 200. - Check long-context output with canary markers rather than status codes alone. - Alert when a long prompt finishes prefill faster than a full prompt should. - Pin SGLang v0.5.22+ and re-check pool size after quantization or parallelism changes. If you run disaggregated inference, verify how your serving engine handles rejected requests before any KV transfer happens. Postmortem: blog.cloudflare.com/kv-cache…

Big win for simple RAG! (raw chat retrieval vs. complex agent memory, clearly explained) A new paper by Zhengye Bao and colleagues shows that dense retrieval with a reranker over raw chat history can outperform complex agent memory systems. The method, called Raw-RAG, ranked first among the compared memory systems in 6 of 7 benchmark-model settings across 4 benchmarks: LoCoMo, LongMemEval, MEME, and MemoryAgentBench. It beat the strongest compared system in each setting by 0.13 to 5.53 points. It also ingested BEAM-1M, a 1,710-message conversation approaching one million tokens, in 0.33 hours, compared with 3.25 hours for MemOS and 26.95 hours for Memora. The reason is how the memory gets stored. Many memory systems rewrite conversations into extracted entries, which can break the links between what changed, when it changed, and what earlier facts it affects. Even complete access to transformed memory stores left an accuracy gap versus raw evidence. Keeping raw history intact preserves those links for retrieval. The diagram below compares Raw-RAG with a memory-extraction pipeline. Here's where that leaves CoALA's 4 memory types, for a coding agent working on your repo for weeks: 𝟭) 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝗺𝗲𝗺𝗼𝗿𝘆 𝗵𝗼𝗹𝗱𝘀 𝘁𝗵𝗲 𝗮𝗰𝘁𝗶𝘃𝗲 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝘄𝗶𝗻𝗱𝗼𝘄. This stores current input, goals, and intermediate reasoning, such as the open file, failing test log, and current fix attempt. Every LLM agent has this by default. 𝟮) 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗺𝗲𝗺𝗼𝗿𝘆 𝘀𝘁𝗼𝗿𝗲𝘀 𝘀𝗲𝗾𝘂𝗲𝗻𝗰𝗲𝘀 𝗼𝗳 𝗽𝗮𝘀𝘁 𝗲𝘃𝗲𝗻𝘁𝘀. Raw-RAG keeps past coding sessions as raw transcripts and searches them directly. Build this first when work spans sessions. 𝟯) 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗺𝗲𝗺𝗼𝗿𝘆 𝘀𝘁𝗼𝗿𝗲𝘀 𝗲𝘅𝘁𝗿𝗮𝗰𝘁𝗲𝗱 𝗳𝗮𝗰𝘁𝘀. Systems often rewrite history into entries like "this repo uses pytest". Add this only when it beats raw retrieval on your evals. 𝟰) 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹 𝗺𝗲𝗺𝗼𝗿𝘆 𝘀𝘁𝗼𝗿𝗲𝘀 𝗿𝘂𝗹𝗲𝘀 𝘁𝗵𝗮𝘁 𝗴𝘂𝗶𝗱𝗲 𝗯𝗲𝗵𝗮𝘃𝗶𝗼𝗿. In a repo, this is your team's PR checklist kept in system prompts or skills. Add this when workflows repeat. Our rule of thumb is to measure raw-history retrieval with a reranker on your own data before adding a memory system. Paper: arxiv.org/abs/2610.30755

We spent months trying to get an LLM to write like a real person while building ToneBench. It broke almost everything we thought we knew about prompting. We started with a banned word list. It didn’t work. When you ban a model’s favorite words, it just grabs the next most predictable synonym. Same rhythm, different word. That’s because next-token prediction drags every line toward the safest average. It’s why AI drafts sit in smooth 18–22 word sentences and end every section with a polite little summary. To track progress, we built an LLM judge. It lied to us. It gave almost every draft a 7 or 8 out of 10, because RLHF taught it to love polite, symmetric paragraphs too. So we killed the 1–10 scale, wrote binary checks for specific AI patterns, and calibrated the judge against human editors. Only then did the scores mean anything. And once we could actually measure it, we found the worst part: newer models write even worse. GPT-6.1 Sol on max effort scored 153 Elo below GPT-6 Sol on our writing benchmark. It also cost $0.13 per script instead of $0.11 and took twice as long. Labs tune releases for math and code, so the LLM-isms keep piling up. What actually worked for us: examples > rules. Instead of banning 30 words, we drop 3 unedited paragraphs from the writer we want to match into context. Then a two-pass audit strips out whatever filler and hedging is left. Our co-founder @Whats_AI is going live with @DataTalksClub on Nov 2 to break down what we learned, and how to set up eval harnesses that catch regressions before your users do. Free to join: luma.com/nassme8x

i use grok bot, claude, openai's dictation everyday. claude seems to understand me the most?

ouch kimi k3.1 posts went south really quick