Alex Gardner

Context engineering? Maybe cache engineering is better? Agents build up a lot of context, and trimming or summarizing older messages can ironically mean more compute as parts of the KV cache need to be recomputed. That's why @Tsinghua_Uni and CUHK propose cache engineering – keep the text history as is, but edit the cache availabililty They introduces 2 operations alongside the normal "append" that adds new KV states to the cache: - "drop" removes selected KV entries from future attention - "repos" adjusts the positional encodings of retained keys The remaining KV states don’t need recomputing, but they can still carry traces of dropped ones, so the order of cache operations matters. ▪️ To bring cache engineering to agents, the researchers built AKV - it's a serving engine that manages cache reuse, eviction, and restoration behind the scenes. It adds several mechanisms that handle the lower-level memory management: 1. A stateless API interface: Agents submit messages and cache instructions without having to manage where cached states physically reside. 2. An extended radix tree - tracks cache states and their operation histories, allowing correct reuse and eviction. 3. Segment-batch attention - reconstructs missing cached states when restoration is necessary, combining work into a single prefill pass. Experiments with a simple policy that gradually drops older tool responses showed: • 27.7% lower average cost • +1.7 points in benchmark scores • 1.18× throughput and 1.51× P95 latency speedup with AKV And letting agents refine their own cache policies cut costs by another 7.1%. But cache engineering necessarily needs specialized serving infrastructure.