[Paper] Towards Interactive Intelligence for Digital Humans

Source: arXiv - 2512.13674v1

Overview

The paper presents Mio (Multimodal Interactive Omni‑Avatar), a new framework that pushes digital humans from static, pre‑scripted avatars toward truly interactive agents. By combining reasoning, natural language, facial and body animation, Mio can express a consistent personality, adapt its behavior on the fly, and even improve itself over time—what the authors call Interactive Intelligence.

Key Contributions

• Interactive Intelligence paradigm – defines a digital human that aligns personality, adapts interactions, and self‑evolves.
• Mio architecture – an end‑to‑end system with five tightly coupled modules:
  1. Thinker (cognitive reasoning & personality modeling)
  2. Talker (context‑aware dialogue generation)
  3. Face Animator (high‑fidelity facial expression synthesis)
  4. Body Animator (gesture and posture generation)
  5. Renderer (real‑time photorealistic visual output)
• Unified multimodal pipeline – all modules share a common latent representation, enabling coherent speech, facial, and body cues.
• New benchmark – a comprehensive evaluation suite that measures personality consistency, interaction adaptivity, visual realism, and self‑evolution capability.
• State‑of‑the‑art performance – Mio outperforms existing digital‑human pipelines on every benchmark dimension.

Methodology

1. Thinker builds a persona graph (traits, goals, memory) using a lightweight transformer that can be updated online.
2. Talker receives the persona state and the dialogue context, then generates responses via a large language model fine‑tuned for consistency and grounding.
3. Face & Body Animators translate the textual output into expressive facial blendshapes and full‑body motion using conditional diffusion models trained on multimodal corpora (speech‑aligned video, motion‑capture).
4. Renderer stitches the animated mesh onto a neural radiance field (NeRF)‑based avatar, delivering photorealistic frames at >30 fps.
5. Self‑evolution loop: after each interaction, feedback signals (user sentiment, task success) are fed back into the Thinker to adjust the persona graph, enabling continual learning without full retraining.

The whole pipeline runs on a single GPU server, making real‑time deployment feasible.

Results & Findings

Human evaluators reported that Mio’s responses felt “more on‑brand” and its gestures “naturally synced” with speech. Ablation studies showed that removing the shared latent space caused a 15 % drop in consistency, confirming the importance of tight multimodal coupling.

Practical Implications

• Customer service bots can now maintain a consistent brand personality while adapting to each user’s tone, reducing churn.
• Virtual training & simulation (e.g., medical, aviation) benefit from avatars that react realistically to trainee actions and evolve based on performance data.
• Gaming & XR developers gain a plug‑and‑play avatar engine that delivers believable NPCs without hand‑crafted animation pipelines.
• Content creation platforms can auto‑generate interview‑style videos where the digital host stays on‑message across multiple episodes.

Because the system runs in real time on commodity hardware, studios and enterprises can integrate it into existing pipelines without massive infrastructure upgrades.

Limitations & Future Work

• Scalability of persona graphs: current Thinker handles a few dozen traits; scaling to richer, long‑term memories may require hierarchical memory structures.
• Data bias: the training corpora are dominated by Western speech and motion patterns, which could limit cultural adaptability.
• Fine‑grained control: while the system is end‑to‑end, designers sometimes need explicit overrides for safety‑critical gestures or speech.
• Future directions suggested by the authors include:
  1. Incorporating multimodal reinforcement learning for more robust self‑evolution.
  2. Expanding the benchmark to cover multilingual and cross‑cultural scenarios.
  3. Optimizing the renderer for mobile AR devices.

Authors

• Yiyi Cai
• Xuangeng Chu
• Xiwei Gao
• Sitong Gong
• Yifei Huang
• Caixin Kang
• Kunhang Li
• Haiyang Liu
• Ruicong Liu
• Yun Liu
• Dianwen Ng
• Zixiong Su
• Erwin Wu
• Yuhan Wu
• Dingkun Yan
• Tianyu Yan
• Chang Zeng
• Bo Zheng
• You Zhou

Paper Information

• arXiv ID: 2512.13674v1
• Categories: cs.CV, cs.CL, cs.GR, cs.HC
• Published: December 15, 2025
• PDF: Download PDF