RedTeam # whyareyouhere

RL and post-training details of Kimi-K3: 1/ i like how Kimi K3's post-training approach decouples capability discovery from capability fusion. instead of requiring a single policy to master all behaviors under one blended RL objective they first develop a high-quality SFT cold-start, train single-task RL experts on diverse behaviors and finally unify the behavior in one model through multi-teacher on-policy distillation (MOPD). as the SFT stage relies on synthetic agent trajectories produced by former domain-specialized Kimi models (which are subsequently verified multi-stage by hand) the base policy is already knowledgeable on how to reason, use tools and perform long-term tasks on trajectories. RL then focuses on learning better execution rather than discovering primitive interaction semantics. finally the MOPD phase presents one practical constraint in specialism: a particular specialization is often helpful in training but deploying multiple individual multi-trillion-parameter checkpoints is impractical both in terms of cost and reliable routing. 2/ there are three domain teacher families in the RL stage: general tasks, general agents and coding agents. general task teacher represents general reasoning and judging ability (which covers vision ability, factual faithfulness, searching ability and knowledge work in professional domain). general-agent teacher represents continued execution ability, which cover long-horizon assistant workflow, deep research and structured writing. coding-agent teacher represents software engineering ability, general coding behavior, GPU kernel optimization, web development, and so on. IMO the choice of these groups is reasonable because Kimi does not train one teacher for each benchmark or skill but train across a distribution of related task and domain. in this way the expert has a coherent behavioral specialization and avoid simply memorize a particular benchmark's format, verifier, tool schema and so on. each domain teacher is trained at three reasoning-effort levels: low, high and max producing nine teachers in total. kimi controls effort using a task-relative token budget rather than assigning the same fixed limit to every problem. 3/ the nine-teacher structure therefore represents a two-dimensional capability matrix: domain specialization × reasoning effort. a coding-max teacher can take more tokens to explore a repository, run tests, fix failed tests and rethink its approach, whereas a coding-low teacher is explicitly optimized to arrive at good output under the constraints of lower output and tool limits. the same would be true for a research and assistant workflows. i think this is a better approach than just training one teacher and adjusting the number of tokens allowed at inference time is that optimal policy itself can change as the budget changes. 4/ MOPD compresses the teachers by choosing the teacher that corresponds to the sampled domain d and effort level e, then evaluating the student on that student's own generated trajectory. for each student-sampled token the teacher provides a clipped log-probability-ratio reward that compares the teacher's probability for that token to the student's. when the teacher's probability for the token is greater than the student's that token receives positive reward, and vice versa for overproduced negatively-rated behavior. The teachers generate evaluations on prefixes of the student's own generated plan so the evaluated states are those the student will run in deployment. 5/ i'd say Kimi K3's RL system is asynchronous for trajectories but not a completely async actor-learner system. they say it augments a standard synchronous RL framework with partial rollouts (N prompts and K sampled responses per prompt). they run NK independent agent-worker trajectories but can start to train policy as soon as λNK's finish rather than waiting for the longest individual episode. they pause aborted trajectories with state preserved (model prefix and sand box) and stick them on a priority queue to have them resumes some later in the learning time. these long agent trajectories may end with very old policies - kimi claims per-token regularization is able to deal with extreme off policy settings although they not disclose total loss. they are able to do this by mixing training and rollouts on same machines, separate KV cache for agents outside the machines for some specific reason, throttling agent on the fly when too large, etc and also making agents checkpointable agentenvs micro VMs. important idea here: Kimi doesn't discard slow individual agent it actually let those long ones across the iteration line.