ai. computers. notes to self.
Joined September 2021
- Tweets18.8K
- Following278
- Followers11.1K
- Likes30.4K
sajal retweeted
difference between max_length and context_length:
1. max_length is a parameter used in LLM API calls, it's an upper cap on your decode (output generation) only, not the prompt. Output stops either at the EOS token or when max_length is hit, whichever comes first. If max_length is 1000 and your output was only 200 tokens, the remaining 800 weren't wasted, no compute wastage, no memory wastage (each new token generated is incrementally added to the KV cache).
Why have max_length at all? Two reasons:
- Cap the output length explicitly, prevent runaway generation
- Indirectly affects concurrency by controlling how long requests hold KV cache pages, longer max_length means requests stay alive in the batch longer, occupying pages longer, leaving less room for new requests
2. context_length is a parameter used when serving LLMs via an inference engine like vLLM or SGLang. It's a ceiling on the total token limit per request, system prompt + input + output combined. The scheduler uses this ceiling when making admission decisions. Actual memory usage is always just what's been generated so far, the ceiling only affects scheduling decisions, not physical page allocation.
It directly impacts concurrency and throughput. Example: global KV cache pool holds 24K tokens worth of pages, context_length is set to 8K, and 3 requests are currently running at 3K tokens each. Each request is physically holding 3K worth of KV cache pages right now, but each could still grow up to 8K before finishing. The scheduler has to account for this worst-case growth, so with 3 requests potentially consuming up to 24K pages total, it can't safely admit new requests until one completes (hits EOS or max_length) and frees its pages back to the pool. This directly throttles throughput.
you rock my world @hkproj !
what a beautiful explanation of HNSW algorithm - youtube.com/watch?v=rhZgXNdh…
565
going from absolute measures to relative measures in anything is a form of greater intelligence i think
1
196
Just started reading about Benchmarks and Evals for OCRs, LLMs, VLMs.
Such a impactful space to be working at right now:
- almost all benchmarks have ground truth errors
- integrity of test eval sets of global benchmarks may have been compromised because many companies/models might use them while competing to do well on these benchmarks - goodhart's law
- still doesn't cover every dimension possible to judge a language model's capability. most of the models can exploit the statistical distribution to get answers on multiple-choice questions that most of these benchmarks have.
we need more unit tests or subjective dimensions to benchmark new language models.
2
127
The more I read about classical architectures (CNN, RNN, LSTM), the more I understand how we came to transformers and how, going from explicit hard-gated regex stuff to more hallucinative black-box stuff, we go towards a general intelligence.
Kind of like we use our brain (hard-gated logic over experiences and rules) plus our real-time intuition(instinct, breaking the rules - hallucinating) in the real-world.
1
129
sajal retweeted
The evaluation pipeline for Indian ASR is broken.
At Sarvam, we kept seeing WER/CER penalize correct output.
They fail on:
• Code-switching
• Script variation (kal vs कल)
• Same meaning, different phrasing
So you get misleading scores → wrong product decisions.
We built better metrics:
LLM-WER, LLM-CER, Intent, Entity
Evaluate Indian ASR in right way using these two frameworks.
We're open-sourcing two frameworks for evaluating Indian ASR, and a full guide on evaluation across 22 languages.
WER (Word Error Rate) and CER (Character Error Rate) were built for languages like English. Indian languages behave differently. Formal and colloquial forms coexist. English words move fluidly between scripts.
In this guide, we introduce four complementary metrics LLM-WER, LLM-CER, Intent Score, and Entity Preservation Score. In combination they capture a more complete view of Indian ASR system performance.
Read more: sarvam.ai/blogs/evaluating-i…
Both frameworks are available to use and build on for evaluating Indian ASR in real-world settings.
Open-source frameworks:
LLM-WER/CER: github.com/sarvamai/llm_wer
Intent + Entity Score: github.com/sarvamai/llm_inte…
insane
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
anthropic.com/glasswing
193
sajal retweeted
The entire assumption that vibe coded applications are inferior comes from assumption that people vibe code because they are bad engineers. The real strength of vibe coding unlocks when you already are a master of entire scope of work being done by AI. Then you do not let bad decisions compound, correct them early, test thoroughly and get to same 1000 concurrent connections significantly quickly.
????
🚨🇫🇷 NEW: The location of the French aircraft carrier, FS Charles de Gaulle, has been given away by a sailor using Strava whilst jogging on the ship deck
[@lemondefr]
1
216
