[Paper] Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos

Source: arXiv - 2512.16907v1

Overview

The paper introduces EgoMAN, a new large‑scale egocentric video dataset and a corresponding model that can predict 3‑D hand trajectories while reasoning about the why behind the motion. By tightly coupling language‑based reasoning with motion generation, the authors bridge the gap between high‑level semantic understanding and low‑level hand control—an ability that can power more intuitive AR/VR interfaces, robotics, and assistive technologies.

Key Contributions

• EgoMAN dataset: 219 K six‑degree‑of‑freedom (6DoF) hand trajectories paired with 3 M structured question‑answer (QA) triples covering semantic, spatial, and motion reasoning across interaction stages.
• Trajectory‑token interface: A novel representation that treats short motion snippets as discrete tokens, enabling seamless integration of language models and motion generators.
• Reasoning‑to‑Motion framework: A two‑stage training pipeline that first aligns vision‑language reasoning with the intended motion, then refines the trajectory generation to respect physical dynamics.
• Stage‑aware prediction: The model can output different trajectories depending on the interaction stage (e.g., reaching, grasping, manipulating), improving realism and task success.
• Cross‑scene generalization: Demonstrated robust performance on unseen real‑world environments, showing the approach scales beyond the training distribution.

Methodology

1. Data collection & annotation

   • Recorded egocentric videos of people interacting with everyday objects (kitchen, office, outdoor).
   • Captured 6DoF hand poses using a calibrated hand‑tracking rig.
   • Annotated each interaction with QA pairs that probe what the hand is doing, why it is moving, and where it will go next.

2. Trajectory‑tokenization

   • Continuous hand motion is split into short, overlapping windows (≈200 ms).
   • Each window is encoded into a discrete token via a learned motion encoder, similar to a visual “vocabulary”.

3. Reasoning module

   • A transformer‑based vision‑language model ingests video frames and the associated QA context, producing a latent “intent” vector.

4. Motion generation module

   • The intent vector conditions a decoder that predicts a sequence of trajectory tokens, which are then de‑tokenized back into smooth 3‑D hand paths using a learned motion decoder.

5. Progressive training

   • Stage 1: Align intent vectors with ground‑truth token sequences (supervised cross‑entropy).
   • Stage 2: Fine‑tune with a dynamics loss (velocity/acceleration consistency) and a stage‑classification loss to enforce stage awareness.

6. Inference

   • Given a new egocentric clip and optional QA prompt, the system outputs a full 6DoF hand trajectory that respects the inferred reasoning and physical feasibility.

Results & Findings

• Semantic grounding: When asked “Why is the hand moving toward the mug?”, the model generated a trajectory that correctly approached the mug’s handle, demonstrating that language cues directly shape motion.
• Generalization: Tested on a held‑out “garage” scene, ADE increased by only 4 mm, indicating robustness to new object layouts.
• Ablation: Removing the reasoning module degraded both accuracy and stage awareness, confirming the importance of the reasoning‑to‑motion link.

Practical Implications

• AR/VR interaction: Developers can embed the model in head‑mounted displays to predict a user’s hand path before the hand is fully visible, enabling smoother object snapping, predictive haptics, and reduced latency.
• Robotics tele‑operation: Translating human intent captured via egocentric cameras into robot hand trajectories can improve remote manipulation in cluttered environments.
• Assistive tech: For users with limited motor control, a reasoning‑aware predictor could auto‑complete hand motions based on high‑level commands (“pick up the pen”).
• Content creation: Animation pipelines can use the model to auto‑generate realistic hand motions from storyboard descriptions, cutting down manual keyframing.
• Dataset as a benchmark: EgoMAN’s QA‑driven structure offers a new way to evaluate models on reasoning as well as precision, encouraging the community to build more cognitively aware motion systems.

Limitations & Future Work

• Hardware dependence: The training data relies on high‑precision hand trackers; scaling to commodity RGB‑only setups may introduce noise.
• Temporal horizon: Current predictions cover up to 2 seconds; longer‑term planning (e.g., multi‑step tasks) remains unexplored.
• Object dynamics: The model assumes static objects; handling deformable or moving objects will require integrating physics simulators.
• Language scope: QA pairs are curated; extending to free‑form natural language commands could broaden applicability.

Future research directions include multimodal fusion with depth/IMU sensors, hierarchical planning for complex task sequences, and open‑domain language grounding to make the system truly conversational.

Authors

• Mingfei Chen
• Yifan Wang
• Zhengqin Li
• Homanga Bharadhwaj
• Yujin Chen
• Chuan Qin
• Ziyi Kou
• Yuan Tian
• Eric Whitmire
• Rajinder Sodhi
• Hrvoje Benko
• Eli Shlizerman
• Yue Liu

Paper Information

• arXiv ID: 2512.16907v1
• Categories: cs.CV, cs.AI, cs.RO
• Published: December 18, 2025
• PDF: Download PDF