Nsak

"A Fix for Softmax and Cross-Entropy" Turns out two of the most fundamental functions in deep learning have numerical bugs that can actually destabilize training. This paper proposes simple fixes to softmax and cross-entropy that prevent tiny gradients from incorrectly rounding to zero, or remaining nonzero when they should vanish, as these errors can cause sudden loss spikes and interfere with knowledge distillation. On small-scale grokking benchmarks, the fixes eliminate Slingshot loss spikes and improve test accuracy from 94.3% to 100% in one setting. They also prevent student models from drifting away from their teachers during distillation, without requiring higher numerical precision. alphaxiv.org/abs/2610.31642