[Paper] PixelGen: Pixel Diffusion Beats Latent Diffusion with Perceptual Loss

Source: arXiv - 2602.02493v1

Overview

PixelGen shows that you can train a diffusion model directly in pixel space and still beat the current state‑of‑the‑art latent diffusion pipelines. By adding two perceptual loss terms that focus on local texture (LPIPS) and global semantics (DINO), the authors guide the model toward a “perceptual manifold” that is easier to learn than the raw high‑dimensional pixel distribution. The result is a simpler, end‑to‑end generator that reaches an FID of 5.11 on ImageNet‑256 without any classifier‑free guidance and with only 80 training epochs.

Key Contributions

• Pure pixel‑space diffusion: Eliminates the VAE encoder/decoder bottleneck used in latent diffusion, removing a major source of artifacts.
• Dual perceptual supervision:
  • LPIPS loss encourages realistic local patterns (textures, edges).
  • DINO‑based loss enforces coherent global semantics (object layout, scene consistency).
• State‑of‑the‑art performance: Outperforms strong latent diffusion baselines on ImageNet‑256 (FID 5.11) and scales well to large‑scale text‑to‑image tasks (GenEval 0.79).
• Training efficiency: Achieves top results with just 80 epochs, far fewer than typical latent diffusion training schedules.
• Open‑source implementation: Code released, facilitating reproducibility and rapid adoption.

Methodology

PixelGen follows the classic denoising diffusion probabilistic model (DDPM) pipeline but operates directly on 256×256 RGB images. The core idea is to replace the naïve pixel‑wise reconstruction loss with two perceptual losses that are computed on intermediate feature maps of pre‑trained networks:

1. LPIPS (Learned Perceptual Image Patch Similarity) – compares deep features from a frozen vision transformer or CNN, penalizing differences in local texture and fine‑grained details.
2. DINO loss – uses features from a self‑supervised DINO model to capture high‑level semantic similarity (e.g., object categories, scene layout).

During training, the diffusion model predicts the noise added to a noisy image at each timestep. The predicted clean image is then fed to both perceptual networks, and the resulting LPIPS and DINO distances are added to the standard diffusion objective. Because the perceptual networks are fixed, they act as a learned, high‑level prior that steers the diffusion process toward perceptually meaningful regions of the pixel manifold, while still allowing the model to learn the full distribution end‑to‑end.

Results & Findings

• Quality: Visual samples show sharper edges, fewer VAE‑induced blurs, and more coherent global composition.
• Efficiency: Faster convergence (80 ×  vs. hundreds of epochs) and no extra encoder/decoder passes.
• Scalability: When scaled to larger text‑conditioned models, the perceptual losses continue to provide a clear advantage, indicating that the approach is not limited to small‑scale benchmarks.

Practical Implications

• Simpler pipelines: Developers can drop the VAE stage entirely, reducing code complexity, memory footprint, and inference latency.
• Faster prototyping: With only a few training epochs needed to reach competitive quality, teams can iterate on model architecture or conditioning strategies more quickly.
• Better integration with downstream tasks: Because the model operates in pixel space, it can be combined directly with other pixel‑level modules (e.g., super‑resolution, inpainting) without needing latent‑space conversions.
• Potential for edge devices: Removing the encoder/decoder halves the number of forward passes, which could make diffusion‑based generation more viable on GPUs with limited VRAM or even on specialized accelerators.
• Open‑source foundation: The released codebase provides a ready‑to‑use template for building custom text‑to‑image or conditional generation systems that benefit from perceptual supervision out of the box.

Limitations & Future Work

• Perceptual loss dependence: The approach relies on pre‑trained LPIPS and DINO models; any bias or limitation in those networks propagates to the generator.
• Memory usage: Operating directly on high‑resolution pixels still demands substantial GPU memory, especially for larger images or batch sizes.
• Generalization to other modalities: The paper focuses on natural images; extending the perceptual‑loss framework to video, 3‑D, or medical imaging remains an open question.
• Ablation depth: While the dual loss shows strong gains, further analysis could reveal whether one loss dominates or if alternative perceptual metrics (e.g., CLIP) provide additional benefits.

Future work may explore lightweight perceptual teachers, mixed‑precision training tricks to lower memory, and applying the same philosophy to multimodal diffusion models (audio‑visual, text‑to‑video, etc.).

Authors

• Zehong Ma
• Ruihan Xu
• Shiliang Zhang

Paper Information

• arXiv ID: 2602.02493v1
• Categories: cs.CV, cs.AI
• Published: February 2, 2026
• PDF: Download PDF