[Paper] Grounding Video Reasoning in Physical Signals

Source: arXiv - 2604.21873v1

Overview

The paper “Grounding Video Reasoning in Physical Signals” presents a new benchmark that pushes video‑question‑answering (VQA) systems beyond surface‑level language tricks. By requiring models to pinpoint what, when, and where a physical event occurs (e.g., a pour, a slide, a collision), the authors expose gaps in current approaches and provide a richer diagnostic for future research.

Key Contributions

• A unified grounded benchmark covering 1,560 clips from four diverse video sources (SSV2, YouCook2, HoloAssist, Roundabout‑TAU).
• Four‑fold evaluation schema (what‑when‑where) that aligns textual queries with explicit temporal and spatial targets.
• Six physics domains (e.g., gravity, friction, momentum) and three prompt families (physics‑focused, V‑STAR‑like, neutral‑restructured) to test semantic robustness.
• Four input perturbations (original, shuffled frames, ablated modalities, frame‑masked) that probe model reliance on visual continuity and physical cues.
• Comprehensive diagnostics showing that (1) physics‑centric prompts are easiest, (2) spatial grounding is the hardest, and (3) robustness varies across prompt families and perturbations.

Methodology

1. Data Unification – Each source video is converted into a grounded event record containing:
   • Semantic label (the “what”)
   • Start/end timestamps (the “when”)
   • Bounding box or region (the “where”)
2. Prompt Generation – From the record three families of natural‑language questions are automatically generated:
   • physics – explicitly mentions physical concepts (e.g., “When does the object start sliding?”)
   • vstar_like – mirrors the original V‑STAR benchmark style, focusing on event description without explicit physics terms.
   • neutral_rstr – templated control questions that are semantically neutral but still require grounding.
3. Model Input Conditions – The same video is presented under four manipulations:
   • Original – untouched video.
   • Shuffled – frames reordered to break temporal continuity.
   • Ablated – certain modalities (e.g., audio or optical flow) removed.
   • Frame‑masked – random frames are occluded.
4. Evaluation – Models are scored on three separate tasks: predicting the correct what label, the correct temporal interval, and the correct spatial region. Accuracy is reported per prompt family and per perturbation, enabling fine‑grained analysis.

Results & Findings

• Physics prompts are the easiest for current models, likely because they contain strong lexical cues that align with training data.
• Spatial grounding is consistently the weakest across all families, indicating that models struggle to localize events precisely.
• Perturbation robustness is selective: models that fail on the original videos sometimes gain modestly when frames are shuffled (suggesting reliance on spurious temporal patterns).
• Prompt‑family robustness does not transfer; a model that excels on physics prompts may falter on neutral_rstr, highlighting the need for prompt‑aware evaluation.

Practical Implications

• More reliable video assistants – Applications such as cooking bots, AR tutoring, or autonomous inspection systems can benefit from models that truly understand when and where an action occurs, not just what it is.
• Safety‑critical monitoring – In robotics or industrial settings, correctly grounding collisions or slips can trigger timely interventions, reducing accidents.
• Benchmark design – The paper’s diagnostic framework encourages developers to report not just aggregate accuracy but also grounding precision and robustness to input noise, leading to more trustworthy AI products.
• Model training strategies – The findings suggest that incorporating explicit spatial supervision (e.g., attention maps, bounding‑box losses) and temporal consistency objectives could close the performance gap on the “where” dimension.

Limitations & Future Work

• Dataset scale – Although diverse, 1,560 clips remain modest compared to large‑scale video corpora; scaling up could reveal new failure modes.
• Domain coverage – The six physics domains are curated; real‑world scenarios may involve more complex, multi‑physics interactions (e.g., fluid‑structure coupling).
• Prompt generation – Automatic templating may miss nuanced linguistic variations that humans naturally use; future work could involve human‑written queries to test linguistic robustness.
• Model diversity – Experiments focus on a handful of existing VQA architectures; exploring transformer‑based video‑language models with dedicated grounding heads is an open avenue.

Authors

• Alibay Osmanli
• Zixu Cheng
• Shaogang Gong

Paper Information

• arXiv ID: 2604.21873v1
• Categories: cs.CV
• Published: April 23, 2026
• PDF: Download PDF