Vivek Iyer

🧵Efficiency Comparisons Lastly, we note that these major improvements do not come at the cost of efficiency – rather, our approach is either comparable or better than token-level decoder-only models, and is significantly more efficient for long-context inputs.

RLVR: REINFORCEMENT LEARNING FROM VERIFIABLE REWARDS Saving the best for the last! The idea in RLVR is to provide a constant reward when there is a mathematically verifiable solution, instead of the scalar reward model output. Everything else is the same - the KL divergence term exists as usual, and then optimization happens with PPO. Key findings: - RLVR is great for targeted math-style tasks (GSM8K, MATH) as well as instruction-following under constraints. Intuitively, in these cases, implementing a verifier (and thus reward) is quite neat. - Next, for RLVR, they find using a general Reward model as the value function, and verifiable rewards in the training objective is the combo that works best, and scalar rewards make optimization noisy. Unfortunately, this also means we still need to train (and load in memory) a separate general-purpose RM) - Starting from a weaker model (like post SFT) works comparably to a DPO model and gets similar amount of verifiable rewards, but KL divergence is higher and test set performance is usually weaker! So, DPO might also be necessary as an initial step, meaning RLVR is not particularly cheap :/