ercem

6 KV cache quantization approaches, explained visually: Quantizing model weights doesn't automatically shrink the KV cache. The cache stores activations generated during inference, and its memory grows with retained tokens and concurrent requests. KV quantization reduces the bytes used to store those activations. For instance, moving from 16-bit to 8-bit halves the raw KV payload, before scales, metadata, and any high-precision buffers. The diagram covers six approaches. In practice, several approaches can be combined within the same implementation. 1. FP8 cache The KV cache normally stores each number using 16 bits. FP8 reduces this to 8 bits, so the numbers take up half the space. A scaling factor adjusts their range before storage. The serving engine needs an attention kernel that supports this format. 2. INT8 / INT4 cache These methods convert cached numbers into 8-bit or 4-bit integers. INT4 uses less memory, but has fewer values to choose from, so rounding introduces more error. During attention, the kernel converts the stored integers back into approximate versions of the original numbers. 3. Quantization granularity Quantization uses a scale to convert numbers into a smaller format. One scale can serve the whole tensor, or separate scales can serve individual channels, tokens, or small groups. Smaller groups fit their numbers more closely. But each group needs its own scale, which also takes memory. 4. KIVI-style quantization Keys and values don't have the same pattern of large numbers, so KIVI compresses them differently. For keys, it groups numbers from the same channel across tokens. For values, it groups numbers within each token. This helps preserve accuracy even at 2-bit precision, without fine-tuning the model. 5. Mixed precision This approach keeps recent tokens' keys and values at higher precision and compresses older entries. Attention reads both parts. Keeping some entries uncompressed avoids rounding errors for those entries, but uses more memory than compressing the entire cache. 6. Non-uniform / outlier-aware quantization Uniform quantization rounds numbers to evenly spaced levels. Non-uniform quantization changes the spacing to better represent the numbers that matter. Outlier-aware methods handle unusually large numbers separately. KVQuant stores these outliers in a separate structure, so they don't force the remaining numbers into a range that's too wide. A smaller cache can fit longer contexts or more requests on the same GPU. It doesn't guarantee faster decoding, because compressing and reading the numbers also takes work. Compare each method against an unquantized cache to check how much memory it saves, whether decoding gets faster or slower, and whether answer quality drops. If you want to dive deeper, I wrote a KV Cache Engineering article covering quantization and other ways to reduce cache memory, with code examples and their trade-offs. Read it below.

Jev for Agent Memory, clearly explained: An agent-memory system often passes conversations, tool results, and task outcomes through an extractor. The extractor identifies possible memories such as user preferences, facts, and procedures. Consider a travel assistant. The user says: - I usually prefer an aisle seat. - For this flight, book a window seat. - The flight is delayed by 25 minutes. These statements should not receive identical treatment. The aisle-seat preference could help with future bookings. The window-seat request applies to one flight. The delay becomes useless once the flight is over. A capable memory system may already handle this distinction. Graphiti, for example, records where a fact came from, resolves it against existing facts, and tracks when it becomes valid or invalid. If the source contains enough context, it can keep the window-seat request specific to one trip. So Jev should not be added to recreate logic the memory system already provides. It becomes useful when the application has its own rules for what may enter long-term memory. A travel application might require that: - Only preferences stated or confirmed by the user become account-level memories. - Preferences inferred from one booking remain attached to that trip. - Passport numbers and payment details never enter general memory. - Ambiguous candidates remain in the session until the user confirms them. These rules require semantic judgment. Once the info has been classified, ordinary code can enforce the policy. The challenge here is deciding whether a sentence describes a lasting preference, a one-time instruction, or weak evidence for an inferred preference. Jev can make that decision before the app even invokes Graphiti. For each candidate, the application sends Jev the proposed fact, its source evidence, and the current user, trip, and session state. It can ask typed questions such as: - Did the user state this preference explicitly? - Does it apply beyond the current trip? - Which retention scope fits it? - Would reusing it on another booking create a meaningful risk? Jev returns choices, scores, and probabilities. Application code applies the thresholds: ↳ Save the aisle-seat preference for the user. ↳ Keep the window-seat request with this trip. ↳ Leave the delay in the active session. ↳ Reject sensitive fields before they reach general memory. Several questions can share the same state in one Jev request. - Approved candidates are then sent to Graphiti as episodes. - From there, Graphiti handles extraction, entity resolution, provenance, temporal updates, and retrieval. GitHub repo: github.com/getzep/graphiti (don't forget to star it ⭐) Note that if Graphiti's existing ingestion behavior meets your application's requirements, the Jev gate is unnecessary. Add Jev when you need to check product-specific rules, such as whether a preference was explicitly confirmed or whether a fact is safe to store, before sending it to Graphiti. If you want to dive deeper, I have also written a hands-on guide showing how to build a Jev-style model with open models, entirely locally. Read it below.

A full-stack LLM engineering roadmap covering: > Prompt engineering > RAG systems > Fine-tuning > Agents > LLM deployment > LLM optimization > Safety, evals & observability > Context engineering Free and open-source resources in the article below. (bookmark this)

The LLM scheduling loop, clearly explained: (bookmark this) Whenever a user sends a prompt to an LLM, the resulting inference request does not go straight from the API endpoint to the GPU. Instead, it first enters the serving engine, where a scheduler decides when and how its tokens will be processed. Here is the complete lifecycle: 1) The request enters the waiting queue Each request arrives with a prompt, sampling settings, and a maximum output length. Prompt lengths vary, and the scheduler cannot know the final generation length in advance. It therefore manages requests one iteration at a time. 2) The scheduler builds the next batch Before every model step, the scheduler checks two main constraints: - How many tokens can be processed in this iteration - How much KV-cache space remains It then applies a scheduling policy, often prioritizing running decode requests before admitting new work. 3) New requests enter prefill During prefill, the model processes the prompt tokens and creates the K and V tensors needed by attention. This stage can process many prompt tokens in parallel. Long prompts may be split into chunks so they do not block active generations for too long. 4) Running requests enter decode Once prefill finishes, the request moves to decode. The model now generates the next token using the KV cache created for all previous tokens. Standard autoregressive decoding usually adds one new token per active request during a model step. 5) The GPU executes the selected work Modern serving engines can place prefill and decode work in the same GPU batch. For example, request A may be processing its prompt while requests B and C generate their next tokens. The batch is therefore not a fixed group that runs until every request finishes. Its composition can change after every model step. 6) The engine updates request state After execution, generated tokens are sampled and appended to their requests. A finished request releases its KV-cache blocks and leaves the batch. An unfinished request keeps its state and becomes eligible for the next scheduling iteration. The scheduler can then use the freed token budget and cache space to admit another waiting request. In the visual below, C finishes after iteration 1, so D enters during iteration 2. By iteration 3, the active batch contains only A and D. This iteration-level replacement is continuous batching. It keeps the GPU working while requests with different prompt and output lengths progress independently. If you want to dive deeper, I wrote a detailed article covering the full LLM inference pipeline, including tokenization, prefill, decode, KV caching, and the latency metrics each stage controls. Read it below.