[Paper] RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation

Source: arXiv - 2601.10606v1

Overview

RSATalker is a new framework that combines the visual fidelity of 3‑D Gaussian Splatting (3DGS) with a “socially‑aware” module to generate realistic talking‑head avatars capable of multi‑turn conversations. By explicitly modeling interpersonal relationships (e.g., family vs. colleague, power dynamics), the system produces video‑level avatars that look and behave more like real people in social VR or virtual assistant scenarios.

Key Contributions

• First socially‑aware talking‑head generator that encodes relationship semantics (blood vs. non‑blood, equal vs. unequal) into the avatar’s facial dynamics.
• Hybrid pipeline: speech‑driven mesh deformation → binding of 3D Gaussians to mesh facets → high‑quality 2‑D rendering, achieving the realism of 3DGS without the heavy compute of large‑scale 2‑D diffusion models.
• Learnable query mechanism for relationship embedding, allowing the model to adapt facial expressions and gaze according to the social context.
• Three‑stage training strategy (mesh motion pre‑training, Gaussian binding, social module fine‑tuning) that stabilizes learning on limited data.
• RSATalker dataset:

  10 k triplets of speech, 3‑D facial mesh, and rendered images, each annotated with relationship labels, released for reproducibility.

Methodology

1. Speech‑to‑Mesh Motion – A lightweight neural network predicts per‑vertex displacements on a canonical facial mesh from the input audio waveform, preserving lip sync and coarse expression.
2. Gaussian Splatting Layer – Each mesh facet is populated with a small set of 3‑D Gaussians whose positions, covariances, and colors are learned. During rendering, the Gaussians are projected onto the screen, producing a photo‑realistic avatar frame in real time (≈30 fps on a consumer GPU).
3. Socially‑Aware Embedding – A set of learnable query vectors attends over a relationship taxonomy (blood/non‑blood, equal/unequal). The resulting embedding modulates the Gaussian attributes (e.g., subtle eye‑contact, head tilt) to reflect the social role of the speaker.
4. Training Pipeline
   • Stage 1: Train the speech‑to‑mesh model on the mesh‑only portion of the dataset.
   • Stage 2: Freeze the mesh model, learn Gaussian parameters to match ground‑truth rendered images.
   • Stage 3: Introduce relationship queries and fine‑tune the whole system end‑to‑end, optimizing a multi‑task loss (lip‑sync, visual realism, relationship consistency).

Results & Findings

• Realism: RSATalker achieves a 0.12 improvement in LPIPS and a 7 % increase in user‑rated visual fidelity over the strongest 3DGS baseline.
• Social Awareness: In a blind study, participants correctly identified the intended relationship (e.g., “talking to a boss” vs. “talking to a friend”) 84 % of the time, compared to 52 % for non‑aware models.
• Efficiency: Rendering a 10‑second clip costs ~0.5 s on an RTX 3060, far cheaper than large‑scale 2‑D diffusion pipelines that require several minutes per frame.
• Ablation: Removing the relationship embedding drops social‑awareness scores by 30 %, confirming its central role.

Practical Implications

• Virtual Reality & Metaverses – Developers can embed RSATalker avatars in social spaces where nuanced interpersonal cues (respectful gaze, subtle posture shifts) improve immersion and reduce the “uncanny valley.”
• Remote Collaboration Tools – Real‑time video avatars that adapt facial behavior based on meeting hierarchy (e.g., presenter vs. attendee) can make virtual meetings feel more natural.
• AI‑Powered Assistants – Customer‑service bots could adjust their facial expressiveness depending on the user’s profile (e.g., more formal with senior executives, more relaxed with peers).
• Game Development – NPCs can exhibit relationship‑driven facial dynamics without hand‑crafting each animation, saving art resources.
• Low‑Cost Production – Because the pipeline runs on consumer GPUs, indie studios and startups can generate high‑quality talking heads without investing in expensive render farms.

Limitations & Future Work

• Dataset Scope – RSATalker’s training data covers a limited set of languages and cultural contexts; performance may degrade on under‑represented accents or gestures.
• Static Backgrounds – Current implementation assumes a fixed background; integrating dynamic environments or full‑body motion remains an open challenge.
• Fine‑Grained Emotion – While relationship cues are captured, subtle emotional states (e.g., sarcasm) are not explicitly modeled. Future work could fuse affective computing signals with the social module.
• Scalability to Large Crowds – Extending the approach to simultaneous multi‑person conversations (group chats) will require more sophisticated interaction modeling.

RSATalker opens the door to socially intelligent, photorealistic avatars that can converse naturally in VR and beyond—an exciting step toward more human‑centric virtual experiences.

Authors

• Peng Chen
• Xiaobao Wei
• Yi Yang
• Naiming Yao
• Hui Chen
• Feng Tian

Paper Information

• arXiv ID: 2601.10606v1
• Categories: cs.CV
• Published: January 15, 2026
• PDF: Download PDF