why does JEPA/DINO-style training produce object-level structure in attention maps with zero segmentation supervision? one specific mechanistic hypothesis keeps coming up: the EMA teacher itself is doing a lot of the actual work DINO's own ablation (Caron et al, arxiv.org/abs/2104.14294) is the cleanest evidence for this: swap the momentum-updated teacher for one that just copies the student's weights directly, and training fails to converge, while if you keeping the EMA, object boundaries emerge in the attention maps on their own, no segmentation labels anywhere in the loop I-JEPA and the rest of the JEPA family inherited the same EMA teacher design for the same reason my read on why this might matter mechanistically: a teacher that changes slower than the student acts as a moving but stable target, so the student can't just chase whatever the teacher did last step, it has to find features that stay predictive across many steps of teacher drift (objects are exactly the kind of feature that stays stable under small transformations while background noise doesn't, so a slow-enough target might be implicitly selecting for object-level structure rather than being told what one is) still an open question how much of this is EMA specifically versus any sufficiently slow, sufficiently stable target (haven't found a paper yet that isolates the two cleanly)