[Paper] SimFlow: Simplified and End-to-End Training of Latent Normalizing Flows

Source: arXiv - 2512.04084v1

Overview

The paper “SimFlow: Simplified and End‑to‑End Training of Latent Normalizing Flows” proposes a surprisingly simple tweak—fixing the VAE’s variance to a constant—to remove the need for noisy data‑augmentation pipelines and to enable joint training of a VAE and a normalizing flow (NF). This change yields state‑of‑the‑art image generation quality on high‑resolution ImageNet while keeping the training pipeline clean and fully end‑to‑end.

Key Contributions

• Constant‑variance trick: Replaces the VAE’s learned variance with a fixed value (e.g., 0.5), eliminating the need for explicit noise injection and denoising steps.
• Joint VAE‑NF training: The simplified ELBO becomes stable enough to train the VAE encoder/decoder together with the NF, removing the common “pre‑train‑then‑freeze” paradigm.
• Performance boost: On ImageNet (256\times256) generation, SimFlow achieves a gFID of 2.15, surpassing the previous best (STARFlow, gFID 2.40).
• Seamless REPA‑E integration: Combining SimFlow with the REPA‑E representation alignment technique pushes gFID further down to 1.91, establishing a new NF benchmark.
• Cleaner pipeline: No extra noise‑generation modules, no separate denoising networks, and a single loss function that covers both reconstruction and flow training.

Methodology

1. Latent VAE backbone – The model uses a standard VAE encoder (E) and decoder (D). Instead of learning a per‑sample variance (\sigma^2), the encoder outputs only a mean vector (\mu). The variance is fixed to a constant (e.g., 0.5).
2. Latent Normalizing Flow – A flow model (F) maps the VAE latent space to a standard Gaussian. Because the latent distribution now has a known, fixed variance, the flow’s log‑determinant term becomes easier to compute and more stable.
3. Unified loss – The training objective combines:
   • The VAE reconstruction loss (pixel‑wise or perceptual) using the decoder on samples drawn from (\mathcal{N}(\mu, 0.5I)).
   • The NF negative log‑likelihood term that pushes the transformed latent to match a unit Gaussian.
     No extra regularizers for noise injection are required.
4. End‑to‑end optimization – Both VAE parameters and flow parameters are updated simultaneously with a single optimizer, simplifying implementation and reducing training time.

Results & Findings

• Quality: Visual samples show sharper textures and better global coherence compared to STARFlow.
• Stability: Training curves indicate smoother convergence, thanks to the constant‑variance ELBO.
• Efficiency: Removing the noise‑generation/denoising modules reduces memory overhead by ~15 % and cuts total training epochs by ~10 %.

Practical Implications

• Simpler pipelines for generative AI: Developers can now integrate a VAE + NF stack without juggling separate augmentation or denoising stages, making codebases easier to maintain.
• Faster prototyping: Joint training means one fewer pre‑training step, shortening the time from research to production.
• Higher‑resolution generation: The gFID improvements at 256 px suggest SimFlow can be a drop‑in replacement for existing NF‑based generators in applications like content creation, data augmentation, and style transfer.
• Compatibility with representation alignment: The fact that SimFlow works out‑of‑the‑box with REPA‑E opens the door to hybrid models that combine the strengths of NFs (exact likelihood) with contrastive or alignment‑based objectives for downstream tasks (e.g., conditional generation, image editing).
• Potential for on‑device deployment: The reduced computational graph and fewer auxiliary networks lower the inference footprint, which is attractive for edge‑AI scenarios where memory is limited.

Limitations & Future Work

• Fixed variance hyper‑parameter: While 0.5 works well empirically, the paper does not explore adaptive or data‑dependent variance schedules, which could further improve reconstruction fidelity.
• Scope limited to image generation: Experiments focus on ImageNet; applying SimFlow to other modalities (audio, video, 3‑D) remains an open question.
• Scalability to ultra‑high resolutions: The study stops at 256 px; it is unclear how the constant‑variance trick behaves when scaling to 1024 px or beyond.
• Theoretical analysis: The authors provide empirical justification but a deeper theoretical understanding of why fixing variance stabilizes the ELBO would strengthen the contribution.

Future work could investigate adaptive variance schemes, extend the approach to multimodal latent spaces, and combine SimFlow with conditional NF architectures for tasks like text‑to‑image synthesis.

Authors

• Qinyu Zhao
• Guangting Zheng
• Tao Yang
• Rui Zhu
• Xingjian Leng
• Stephen Gould
• Liang Zheng

Paper Information

• arXiv ID: 2512.04084v1
• Categories: cs.CV
• Published: December 3, 2025
• PDF: Download PDF