Datou

Excited to finally share C2R#! 🚀 Coarse-to-Real (C2R)# turns rough 3D and anime videos into photorealistic videos, without requiring paired video training data, by learning from much cheaper image pairs instead. Last year, we introduced C2R, using video generation models to turn rough 3D blockouts into photorealistic videos. But making this work reliably in a game engine is another story. Even models as powerful as Seedance 2.5 often require multiple attempts to get the desired result. For real applications, we need stable and controllable generation. This typically requires large amounts of paired coarse/real videos, which are extremely expensive to collect. So how can we get around this data bottleneck? One evening over dinner, while I was struggling with this problem, our intern Daohan proposed an interesting idea: Could we learn video translation from image pairs instead? After all, image pairs are much easier and cheaper to obtain. Over the following months, we worked hard to turn this idea into reality. We developed a lightweight Real-to-Coarse model trained only on image pairs. By applying it frame by frame to real videos, we can synthesize pseudo-coarse videos and create large amounts of pseudo-paired video training data. We then use these pairs to train a large Coarse-to-Real video generation model, without needing actual paired coarse/real videos! The resulting C2R# technology was eventually used at Roblox RDC. C2R# can also handle other input proxies, including anime-style videos, as long as you have 10K image pairs. 🌐 Project: gonzalognogales.github.io/co… 📄 Paper: arxiv.org/abs/2610.28954