[Paper] Creative Image Generation with Diffusion Model

Source: arXiv - 2601.22125v1

Overview

The paper introduces a fresh way to coax diffusion‑based text‑to‑image models into producing creative outputs—images that are both high‑quality and unlikely to exist in the usual CLIP embedding space. By steering the generation process toward low‑probability regions, the authors achieve strikingly novel visuals without sacrificing realism, opening a new avenue for AI‑assisted imagination.

Key Contributions

• Creativity metric via inverse CLIP probability: Defines creativity as the inverse likelihood of an image’s embedding under a CLIP‑trained distribution.
• Probabilistic steering of diffusion models: Implements a loss that pushes generated samples into low‑density zones of the CLIP space, encouraging rare concepts.
• Pullback mechanisms: Introduces corrective steps that pull samples back toward the data manifold, preserving visual fidelity while maintaining high creativity.
• Unified framework: Works with off‑the‑shelf text‑to‑image diffusion models (e.g., Stable Diffusion) without hand‑crafted prompt engineering or concept blending.
• Extensive empirical validation: Demonstrates that the method consistently yields more novel and thought‑provoking images across multiple benchmarks.

Methodology

1. Embedding‑Space Density Estimation – A pre‑trained CLIP model maps any image to a high‑dimensional embedding. The authors fit a simple density estimator (e.g., a Gaussian Mixture Model) on embeddings of a large image corpus to obtain a probability density function (p_{\text{CLIP}}(z)).
2. Creativity Loss – During diffusion sampling, an auxiliary loss term (\mathcal{L}{\text{crea}} = -\log p{\text{CLIP}}(z_t)) is added, where (z_t) is the current latent’s CLIP embedding. Minimizing this loss pushes the latent toward regions where (p_{\text{CLIP}}) is low (i.e., “rare” embeddings).
3. Pullback Step – After each diffusion step, a small corrective update nudges the latent back toward the learned diffusion manifold using the standard denoising score. This prevents the sample from drifting into unrealistic artifacts.
4. Integration with Existing Pipelines – The creativity loss is applied on top of the usual classifier‑free guidance, requiring only a few extra forward passes through CLIP per diffusion timestep, making the approach compatible with existing inference pipelines.

Results & Findings

• Quantitative novelty: Measured by KL‑divergence between generated embeddings and the training distribution, the proposed method achieves a 2–3× increase over baseline diffusion sampling.
• Visual fidelity: FID scores remain comparable to the original model (ΔFID < 0.05), confirming that pullback mechanisms successfully retain image quality.
• Human evaluation: In a blind study with 200 participants, 78 % of the creative samples were rated as “more imaginative” than baseline outputs, while 85 % were still considered “plausible.”
• Efficiency: The added CLIP forward passes increase inference time by ~15 %, a modest overhead given the gain in novelty.

Practical Implications

• Design & advertising: Brands can generate eye‑catching concepts (e.g., product mock‑ups, campaign art) that stand out from the usual AI‑generated stock images.
• Game development & VFX: Artists can explore unconventional textures, creatures, or environments without manually crafting prompts for each variation.
• Rapid prototyping: Developers building creative assistants (e.g., AI‑powered brainstorming tools) can embed the creativity loss to suggest truly fresh visual ideas.
• Content moderation & safety: By understanding low‑probability regions, platforms can better anticipate novel, potentially problematic content before it proliferates.

Limitations & Future Work

• Density estimator simplicity: The current Gaussian‑Mixture model may not capture complex multimodal structures in CLIP space, limiting the granularity of “creativity.”
• Computational overhead: Although modest, the extra CLIP passes could be prohibitive for real‑time mobile applications.
• Subjectivity of creativity: The inverse probability metric is a proxy; future work could incorporate user‑feedback loops or multimodal novelty measures.
• Cross‑modal extensions: Applying the same principle to video or 3‑D asset generation remains an open research direction.

Bottom line: By reframing creativity as a probabilistic pursuit in CLIP’s embedding world, this work equips developers with a principled, plug‑and‑play tool to push diffusion models beyond the familiar and into the truly imaginative.

Authors

• Kunpeng Song
• Ahmed Elgammal

Paper Information

• arXiv ID: 2601.22125v1
• Categories: cs.CV
• Published: January 29, 2026
• PDF: Download PDF