[Paper] GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures

Source: arXiv - 2512.09925v1

Overview

GAINS tackles a surprisingly common problem in 3‑D reconstruction: extracting reliable geometry, material properties, and lighting from only a handful of photographs. By marrying Gaussian‑splatting‑based inverse rendering with learned depth, normal, and diffusion priors, the authors deliver a system that remains robust when the input views are sparse—something previous state‑of‑the‑art methods struggle with.

Key Contributions

• Two‑stage inverse‑rendering pipeline that first stabilizes geometry with monocular depth/normal priors, then refines material estimates using segmentation, intrinsic image decomposition (IID), and diffusion priors.
• Sparse‑view resilience: Demonstrates a >30 % boost in material‑parameter accuracy and noticeably better relighting quality when only 3–5 views are available.
• Unified Gaussian representation: Extends the popular Gaussian splatting framework with physically‑based shading parameters while keeping the rendering pipeline fully differentiable.
• Extensive benchmark: Provides quantitative and qualitative results on both synthetic datasets (BlenderProc, DTU) and real‑world captures (handheld phones), establishing new baselines for sparse multi‑view inverse rendering.
• Open‑source release: Code, pretrained models, and an interactive demo are made publicly available, encouraging rapid adoption and further research.

Methodology

Stage 1 – Geometry Stabilization

1. Starts from a coarse Gaussian‑splatting reconstruction built from the sparse views.
2. Injects learning‑based priors: a monocular depth network (MiDaS‑style) and a normal estimator provide per‑pixel cues; a diffusion model supplies a global shape prior that discourages unrealistic surface folds.
3. These cues are fused into a joint loss that refines the positions and covariances of the Gaussians, yielding a more plausible geometry without requiring dense coverage.

Stage 2 – Material & Lighting Recovery

1. The refined geometry is fixed while the system optimizes reflectance (diffuse albedo, specular roughness, etc.) and illumination.
2. Segmentation masks isolate objects, reducing cross‑talk between different materials.
3. An intrinsic image decomposition network supplies an initial guess for albedo vs. shading, acting as a strong regularizer.
4. A diffusion prior on material maps (trained on a large collection of BRDF textures) encourages realistic spatial smoothness and plausible texture statistics.
5. All components are optimized end‑to‑end using a differentiable renderer that evaluates the photometric error between rendered and captured images.

The whole pipeline runs on a single GPU and converges in a few minutes for typical 3‑view inputs, making it practical for developers.

Results & Findings

• Geometry: Mean Chamfer distance improves by ~20 % in sparse settings, confirming that the depth/normal priors effectively resolve ambiguities.
• Material Accuracy: Albedo and roughness errors drop dramatically, leading to more faithful texture recovery.
• Relighting & Novel View Synthesis: Rendered images under new lighting conditions are noticeably cleaner, with fewer ghosting artifacts and better specular handling.
• Ablation: Removing the diffusion prior on materials degrades RMSE by ~15 %, highlighting its role in enforcing realistic texture statistics.

Practical Implications

• Rapid Asset Creation: Game studios and AR developers can generate high‑quality 3‑D assets from just a few smartphone photos, cutting down on costly photogrammetry sessions.
• Virtual Try‑On & E‑Commerce: Accurate material recovery enables realistic product visualizations (e.g., fabric sheen, metal gloss) without exhaustive studio lighting rigs.
• Robotics & Scene Understanding: Sparse‑view inverse rendering can enrich SLAM pipelines with material cues, improving grasp planning and illumination‑aware navigation.
• Content‑Driven Relighting: Post‑capture lighting edits become feasible for creators who only have limited reference images, opening new workflows in VFX and digital twins.

Because GAINS builds on the already popular Gaussian splatting ecosystem (e.g., Nerf‑Gaussian‑Splatting), integrating it into existing pipelines requires minimal code changes—mostly swapping the optimizer and adding the prior modules.

Limitations & Future Work

• Dependence on Pretrained Priors: The quality of depth, normal, and diffusion priors directly influences the final result; failure cases arise when these networks encounter out‑of‑distribution scenes (e.g., heavy translucency).
• Static Lighting Assumption: GAINS assumes a single, static illumination environment per capture set; dynamic lighting or mixed‑lighting scenes are not yet supported.
• Scalability to Large Scenes: While efficient for object‑scale captures, extending the method to whole‑room reconstructions will require hierarchical Gaussian management and more memory‑friendly priors.
• Future Directions: The authors plan to explore self‑supervised prior refinement, incorporate temporal priors for video capture, and experiment with neural‑field‑based lighting representations to handle complex illumination.

Authors

• Patrick Noras
• Jun Myeong Choi
• Didier Stricker
• Pieter Peers
• Roni Sengupta

Paper Information

• arXiv ID: 2512.09925v1
• Categories: cs.CV
• Published: December 10, 2025
• PDF: Download PDF