darren

Great post from Base Labs on the impact of SFT with teacher-generated trajectories before RL. The key findings are as follows… Setup. They train 0.6B, 1.7B, 4B, and 8B Qwen models with either SFT + RL or just RL. SFT traces are generated by GPT-5.6-Sol and include both reasoning and final answers, produced by simply prompting the teacher to explain its reasoning. They vary the amount of SFT data—0.5M, 1M, 2M, or 4M tokens. Training and evaluation use 16 synthetic calculation and structured reasoning tasks from Reasoning Gym. Key findings. The headline findings from this analysis: - SFT improves base model accuracy before RL across all model sizes. - Improving accuracy before RL does not always lead to better performance after RL. - Larger models (4B and 8B) see little or no benefit from SFT before RL, while smaller models (0.6B and 1.7B) perform better with SFT + RL. - Increasing the amount of SFT data does not change these trends—the impact of scaling SFT is inconsistent across model sizes. Together, these results show that pre-RL performance is not necessarily a good indicator of final performance after RL. SFT improves initial accuracy, but some models—especially larger models—achieve better final performance with RL alone. What causes these trends? Given these findings, we might wonder why SFT helps some models but hurts others. The answer boils down to two factors: 1. Can the model already solve some problems before RL? 2. How does SFT impact the entropy / diversity of model outputs? For GRPO to provide a useful learning signal, the model must produce a mixture of correct and incorrect responses to each prompt. However, smaller models struggle to solve many of these problems. By running SFT, we can teach these models to produce some correct solutions, enabling RL to make progress. Larger models, however, are often already capable of finding correct solutions when sampling multiple completions. Running SFT does not significantly improve this ability, but it does measurably reduce next-token entropy—making model outputs less diverse and limiting exploration. As a result, SFT provides little benefit to larger models and may even hinder RL training. Takeaway. For RL to work well, the model must be capable of solving problems in at least a subset of its rollouts. If the model cannot produce successful solutions, RL will progress slowly, if at all. SFT can address this by improving initial capabilities and enabling RL to make progress. However, SFT also reduces next-token entropy, potentially limiting exploration during RL. Therefore, we should only run SFT when necessary. Larger models may already solve enough problems for RL to make progress, meaning additional SFT provides little benefit while reducing output diversity.