[Paper] PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes

Source: arXiv - 2512.24986v1

Overview

PhysTalk introduces a novel pipeline that lets users turn plain‑language prompts into real‑time, physics‑driven animations of 3D Gaussian Splatting (3DGS) scenes. By harnessing a large language model (LLM) to generate executable code that directly manipulates 3DGS parameters and particle dynamics, the system bypasses costly mesh extraction and offline rendering, opening the door to interactive, “talk‑to‑your‑scene” experiences for developers and creators.

Key Contributions

• LLM‑driven code generation that translates arbitrary text prompts into executable physics instructions for 3DGS scenes.
• Direct coupling of 3D Gaussian splatting with a physics simulator (no intermediate mesh conversion), enabling collision‑aware, multi‑material dynamics in real time.
• Train‑free, lightweight architecture that runs on commodity GPUs, shifting animation from a batch “render‑and‑wait” workflow to an interactive dialogue.
• Open‑vocabulary support, allowing users to describe novel objects, forces, or actions without pre‑defining a fixed set of commands.
• Demonstrated interactive 4D (space + time) editing, where users can iteratively refine animations through natural language.

Methodology

1. Input Representation – The scene is stored as a 3D Gaussian Splatting model, a compact collection of Gaussian primitives that encode geometry, appearance, and opacity.
2. Prompt Parsing – A large language model (e.g., GPT‑4) receives the user’s textual instruction (e.g., “make the red ball bounce off the floor”) and generates a short Python‑like script.
3. Proxy Layer – The generated script calls a thin “proxy” API that maps high‑level commands to low‑level 3DGS parameter updates (e.g., position, scale, material) and to particle‑based physics primitives (rigid bodies, soft bodies, forces).
4. Physics Integration – A lightweight particle dynamics engine (e.g., Position‑Based Dynamics) simulates collisions, gravity, and constraints directly on the Gaussian primitives, updating their attributes each frame.
5. Real‑time Rendering – The updated Gaussian parameters are fed back into the 3DGS renderer, producing a smooth, view‑consistent animation at interactive frame rates (≈30–60 fps on a modern GPU).
6. Iterative Loop – Users can issue follow‑up prompts; the LLM re‑generates or patches the script, enabling a conversational editing cycle.

Results & Findings

• Speed – PhysTalk achieves interactive rates (≈30 fps) for scenes with up to ~1 M Gaussians, far faster than mesh‑based pipelines that require seconds to minutes of offline simulation.
• Physical Plausibility – Qualitative demos show convincing rigid‑body collisions, bouncing, stacking, and soft‑body deformations across multiple materials without any pre‑training.
• Open‑Vocabulary Success – The system correctly interprets novel object descriptors (“glowing crystal”, “rubber duck”) and applies appropriate physics parameters (e.g., density, restitution) derived from LLM knowledge.
• User Study – A small informal study (n = 12) reported a 4.2/5 average satisfaction score for “ease of creating desired animations” compared to traditional key‑frame tools.
• Resource Footprint – The entire pipeline runs on a single RTX 3080 with < 2 GB VRAM overhead beyond the base 3DGS model.

Practical Implications

• Game & AR/VR Prototyping – Designers can rapidly prototype interactive physics effects (e.g., explosions, ragdoll responses) without writing shader code or baking simulations.
• Content Creation Platforms – Cloud‑based editors could expose a “talk‑to‑your‑scene” interface, letting non‑technical artists animate assets on the fly.
• Simulation‑as‑a‑Service – Engineers can generate quick physics previews for CAD models or robotics scenarios by simply describing constraints in natural language.
• Education & Training – Interactive physics demos become accessible to students who can ask “What happens if I drop a glass bottle on a wooden table?” and instantly see the result.
• Reduced Pipeline Complexity – By eliminating mesh extraction and separate physics preprocessing, development pipelines become leaner, cutting down on storage, licensing, and maintenance costs.

Limitations & Future Work

• Physics Fidelity – The particle‑based engine trades realism for speed; high‑precision contact modeling (e.g., friction anisotropy) is still limited.
• Complex Topologies – Extremely intricate geometries may require a higher Gaussian count, which can strain real‑time performance.
• LLM Hallucinations – Occasionally the generated script misinterprets ambiguous prompts, leading to unintended forces or parameter values.
• Scalability to Multi‑User Scenarios – Synchronizing physics state across networked participants remains an open challenge.
• Future Directions – The authors plan to integrate more advanced differentiable physics, explore fine‑tuned LLMs for domain‑specific vocabularies, and evaluate large‑scale user studies in production‑grade tools.

Authors

• Luca Collorone
• Mert Kiray
• Indro Spinelli
• Fabio Galasso
• Benjamin Busam

Paper Information

• arXiv ID: 2512.24986v1
• Categories: cs.GR, cs.CV
• Published: December 31, 2025
• PDF: Download PDF