[Paper] M*: A Modular, Extensible, Serving System for Multimodal Models
Source: arXiv - 2606.12688v1
Overview
We are entering a new era of composite model architectures that integrate diverse components such as vision encoders, language backbones, diffusion and flow heads, audio codecs, action generators, and world-model predictors. Such architectures underpin a broad class of multimodal models, including unified multimodal models, omni models, speech-language models, vision-language-action policies, and world models. However, existing model serving frameworks were built on narrow assumptions about model structure, making them ill-suited to accommodate this new architectural diversity. Here we present M*, a universal serving system for efficient serving of composite AI models. M* represents models as dataflow graphs, processing requests spanning diverse modalities and tasks as traversals over these graphs. The core insight is a modular abstraction that supports arbitrary composition of model components, flexible placement onto a physical cluster, and model-agnostic optimizations within a distributed runtime. We call this abstraction the Walk Graph and show how it can concisely capture composite models from a broad range of families. We instantiate M* on representative models and find that it achieves, on average, 20% lower end-to-end latency than vLLM-Omni for text-to-image workloads on BAGEL, while delivering up to 2.9x lower real-time factor and 2.7x higher throughput for text-to-speech workloads on Qwen3-Omni. M* also outperforms the V-JEPA 2-AC rollout baseline for robotic planning by up to 12.5x. Thus, our work paves the road towards more efficient serving of complex models with minimal developer effort.
Key Contributions
This paper presents research in the following areas:
- cs.LG
- cs.AI
- cs.DC
Methodology
Please refer to the full paper for detailed methodology.
Practical Implications
This research contributes to the advancement of cs.LG.
Authors
- Atindra Jha
- Naomi Sagan
- Keisuke Kamahori
- Irmak Sivgin
- Rohan Sanda
- Steven Gao
- Mark Horowitz
- Luke Zettlemoyer
- Olivia Hsu
- Jure Leskovec
- Baris Kasikci
- Stephanie Wang
Paper Information
- arXiv ID: 2606.12688v1
- Categories: cs.LG, cs.AI, cs.DC
- Published: June 10, 2026
- PDF: Download PDF