Jamie Dougherty

What does it take to scale self-distillation across many GPUs? Self-distillation replaces the fixed, pre-trained teacher in standard knowledge distillation with a teacher that is an exponential moving average (EMA) of the student’s weights: - The student is trained with gradient descent - The teacher never backprops and is updated only via EMA of the student Over time, the EMA teacher becomes an implicit ensemble of past students (more stable and less noisy). This provides higher-quality targets that let the student improve in a continual bootstrapping loop. But once you try to distribute this setup, you run into a structural headache – student's and teacher's updates need to stay perfectly in sync. @Speechmatics tested 3 strategies in production while scaling self-distillation: 1. DDP: Full student + teacher on every GPU 2. FSDP (student only): Sharded student, replicated teacher 3. FSDP (both): Student and teacher sharded identically across GPUs Their main practical finding: distributed training has to match the shape of the algorithm. Here is more on how each strategy works and why identical FSDP sharding of the student and teacher is the best setup for self-distillation. ⟶ blog.speechmatics.com/distir… (by @edjong3 & Timofey Abramski) It’s really worth exploring to get the most out of compute and speed.