Reinforcement learning (RL), especially reinforcement learning with verifiable rewards, has become an essential technique for improving reasoning, safety, and alignment in large language models. Historically, practitioners have relied on the AdamW optimization algorithm because it is the standard for pretraining and supervised fine-tuning, despite its heavy memory footprint. However, RL introduces a fundamentally distinct learning environment where models sample their own data on the fly and receive sparse feedback. This non-stationary process raises the question of whether the complex, memory-intensive mechanisms within AdamW—specifically momentum and adaptive per-parameter learning rates—are genuinely necessary for RL. The article evaluated whether basic Stochastic Gradient Descent (SGD), which tracks no historical optimizer states and uses substantially less memory, can match or exceed the performance of AdamW in reinforcement learning fine-tuning. The authors investigated the individual contributions of momentum and adaptive learning rates across multiple model families, tasks, and reinforcement learning algorithms. To test this, the researchers conducted extensive empirical experiments using Qwen (1.7-billion and 8-billion parameters) and Llama (8-billion parameters) models across mathematical reasoning, competitive programming, and synthetic coding benchmarks. They tested two reinforcement learning frameworks—Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO)—and systematically ablated optimizer components by comparing standard AdamW, RMSProp (adaptive rates without momentum), SGD with momentum (momentum without adaptive rates), and vanilla SGD. Read this paper with an AI tutor: chapterpal.com/s/31ab944b/do…