[Paper] ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control

Source: arXiv - 2604.20816v1

Overview

The paper presents ParetoSlider, a new way to fine‑tune diffusion models (the backbone of many modern image generators) so that a single trained model can be steered at inference time along the entire spectrum of competing objectives—e.g., how closely an edited image follows the user’s prompt versus how faithfully it preserves the original content. By treating the reward weights as a conditioning variable during training, the authors let developers “slide” between trade‑offs without retraining or swapping checkpoints.

Key Contributions

• MORL‑enabled diffusion training: Introduces a multi‑objective reinforcement‑learning (MORL) framework that learns the full Pareto front for diffusion models.
• Preference‑conditioned conditioning: Uses a continuous scalar (or vector) representing reward weights as an extra input to the diffusion model, enabling on‑the‑fly adjustment of objectives.
• Single‑model solution: Achieves performance comparable to—or better than—separate models trained for each fixed trade‑off, cutting storage and maintenance overhead.
• Broad backbone compatibility: Demonstrates the approach on three state‑of‑the‑art flow‑matching backbones (SD3.5, FluxKontext, LTX‑2), showing it is not tied to a specific architecture.
• Empirical validation: Provides quantitative and qualitative evidence that ParetoSlider can navigate between prompt adherence and source fidelity, as well as other conflicting criteria, with smooth, predictable behavior.

Methodology

1. Define multiple rewards – The authors pick two (or more) reward functions that capture the competing goals (e.g., CLIP‑based prompt similarity vs. structural similarity to the input image).
2. Preference vector as conditioning – During training, a random preference weight λ ∈ [0,1] (or a higher‑dimensional weight vector) is sampled and concatenated to the diffusion model’s conditioning inputs (e.g., text prompt, latent). This tells the model “how much to care about each reward this round.”
3. MORL loss – The standard diffusion loss is augmented with a reinforcement‑learning style policy gradient term that maximizes the weighted sum λ·R₁ + (1‑λ)·R₂. Because λ changes each step, the model sees the whole continuum of trade‑offs.
4. Training loop – The model is trained on a large dataset of image‑prompt pairs (or image‑to‑image edits) using the usual diffusion objective plus the MORL term. No extra checkpoints are saved; the single network learns to map any λ to the appropriate generation behavior.
5. Inference slider – At generation time, developers simply set the desired λ (or a slider UI) and run the diffusion process. The model produces outputs that lie on the learned Pareto front for that weight configuration.

Results & Findings

• Smooth control: Varying λ produces a monotonic shift in both metrics, confirming that the model learned a coherent Pareto front.
• No performance penalty: Even at extreme ends (λ ≈ 0 or 1), ParetoSlider’s outputs are on par with models trained exclusively for those single objectives.
• Qualitative examples: Side‑by‑side images show how increasing λ yields more aggressive prompt‑driven edits, while decreasing λ preserves more of the original image’s structure.

Practical Implications

• Single‑model deployment: Companies can ship one diffusion checkpoint that serves multiple use‑cases (creative generation, faithful editing, style transfer) simply by exposing a UI slider.
• Reduced storage & CI costs: No need to maintain a fleet of checkpoints for each reward weighting; updates affect all trade‑offs simultaneously.
• Dynamic user personalization: End‑users can fine‑tune the balance between creativity and fidelity in real time, leading to better satisfaction in photo‑editing apps, generative design tools, and AI‑assisted content creation platforms.
• Rapid prototyping: Researchers can experiment with new reward combinations (e.g., adding a safety or bias‑mitigation term) without retraining from scratch—just augment the preference vector.
• Potential for API services: Cloud providers can expose a “ParetoSlider” parameter in their generation endpoints, giving developers a simple knob to meet diverse SLAs (speed vs. quality, novelty vs. consistency).

Limitations & Future Work

• Scalability to many objectives: The paper focuses on two competing rewards; extending to three or more may require higher‑dimensional conditioning and could complicate the slider UI.
• Reward design dependency: The quality of the Pareto front hinges on well‑behaved, differentiable reward functions; noisy or poorly calibrated rewards could destabilize training.
• Computational overhead: Adding the MORL policy‑gradient term modestly increases training time compared to vanilla diffusion training.
• Generalization to non‑image domains: While demonstrated on image diffusion, applying ParetoSlider to text, audio, or multimodal generators remains an open question.
• Future directions: The authors suggest exploring adaptive preference sampling (focus training effort on under‑represented regions of the front), integrating user feedback loops for online fine‑tuning, and scaling to large‑scale foundation models with dozens of objectives.

Authors

• Shelly Golan
• Michael Finkelson
• Ariel Bereslavsky
• Yotam Nitzan
• Or Patashnik

Paper Information

• arXiv ID: 2604.20816v1
• Categories: cs.LG, cs.CV
• Published: April 22, 2026
• PDF: Download PDF