Lazar Atanackovic

i am extremely excited to announce our latest flow map finetuning algorithm: Wasserstein Tilted Flows (WTF?!) something i’ve been thinking about for a while now is the “right” way to formulate finetuning and reward alignment of flow-based generative models. two difficulties that arise repeatedly are 1) sampling the canonical “reward-tilted distribution” is really hard, and 2) basically all existing methods require backpropagation or adjoint methods through flow rollouts, which is prohibitively expensive the core issue i’ve had with the reward tilt is that it arises from a geometry on the space of probability measures based on the KL divergence. this makes sense for stochastic processes such as diffusions, but for the deterministic flows we use in practice, this choice of geometry can be very awkward, leading state of the art methods to repeatedly convert back and forth between a flow and a diffusion here we throw out this formulation entirely, redefining the reward alignment problem in terms of a carefully-chosen wasserstein distance. we show that that this leads to a deterministic optimal control problem, which we can solve rapidly via reinforcement learning in a simulation-free by way leveraging the underlying flow map. critically, using the flow map avoids backpropagation through rollouts, replacing them by a single function evaluation. by adapting both the flow map and its underlying flow together online, we obtain an end-to-end finetuning algorithm that takes a flow map in, gives a flow map out, and is ~50-100x faster than current state of the art approaches super proud of my students for their hard work on this one — check it out!