Paper Instruments

check out github.com/paper-instruments… to see how we use verifiers as a portable rollout engine across training frameworks. the repo includes a reference tinker integration, but standardizing on verifiers made it very easy to integrate with miles, other tinker compatible apis, and prime-rl for experiments.

found out with the @paperinstr team that hash based session routing for agent rl rollouts can deathspiral your samplers and kill your job's throughput. for max sampling throughput, you generally want to operate close to the kv cache capacity. leaving too much headroom wastes capacity, but crossing the limit causes evicted rollout prefixes to be re-prefilled. your job's max concurrent rollouts per replica is lower bounded by kv cache capacity: max rollouts / replica >= kv cache capacity / kv bytes at max context length in practice, each rollout's working set is usually much smaller than the max context. if average rollout residency is ~50% of max context, you can schedule maybe ~2x that (with headroom) agent sessions with cumulative context are typically pinned to sampling replicas to maximize prefix cache hits. standard inference engines have no session awareness, so during environment interactions, a rollout's cached kvs become eligible for eviction. if the engine has a request queue from other concurrent rollouts and is under cache pressure, the engine will admit the new prefill and evict the cache of any "idle" rollouts. hash based routing (e.g. vllm router's consistent_hash policy) is a popular load balancing choice for sampling replicas. hash based routing produces uniform session distribution in expectation, but does not guarantee perfectly balanced live session counts. skew creates overloaded and underloaded replicas, and an overloaded replica can become "poisoned" with no escape valve. when rollouts on underloaded replicas finish, their replacement can hash to an already overloaded one. cache pressure -> eviction -> re-prefill -> lower throughput -> bigger queue -> more cache pressure a simple mitigation is using session-based least loaded assignment. new sessions are assigned to the replica with the least number of active sessions. if you conservatively cap in-flight rollouts to the per replica lower bound described above, least loaded placement entirely prevent eviction. if you intentionally oversaturate based on expected residency, even though eviction is possible, least loaded placement give the useful property that the replica under pressure becomes quarantined. new rollouts stop being scheduled on the replica under pressure until one of its existing sessions finishes, allowing the system to recover. perfect least loaded session routing does require the router to know when a trajectory is complete so it can decrement its live session count. many generic inference routers don't expose a "release session" api because they are designed around requests, not sessions. a throughputmaxxing rollout scheduler should dynamically change rollout concurrency, pause rollouts, or explicitly offload/evict cache by session based on kv cache pressure and environment interaction time. this is especially important for rl where trajectories can grow over time as the policy changes, or when tasks have substantially different length distributions.