[Paper] Chart2Code-MoLA: Efficient Multi-Modal Code Generation via Adaptive Expert Routing

Source: arXiv - 2511.23321v1

Overview

The paper Chart2Code‑MoLA tackles the long‑standing problem of turning visual charts (bar graphs, line plots, heatmaps, etc.) into runnable code that reproduces the same visualization. By marrying a Mixture‑of‑Experts (MoE) routing scheme with Low‑Rank Adaptation (LoRA), the authors deliver a model that is both more accurate on diverse chart types and lighter on GPU memory—key concerns for developers building automated data‑visualization pipelines.

Key Contributions

• Adaptive expert routing: A complexity‑aware gate directs each chart to one of several domain‑specialized experts (e.g., simple bar charts vs. dense heatmaps).
• Parameter‑efficient fine‑tuning: LoRA adapters (rank‑8) are added on top of a frozen backbone, cutting the number of trainable parameters dramatically.
• Joint training strategy: The authors synchronize routing stability with semantic loss, ensuring that the same expert consistently handles similar inputs.
• Empirical gains: On the large‑scale Chart2Code‑160k benchmark, the model achieves up to +17 % generation accuracy, ‑18 % peak GPU memory, and ‑20 % convergence time versus standard fine‑tuning or LoRA‑only baselines.
• Scalable design: Ablation studies identify 8 experts as the sweet spot, confirming that the architecture scales without exploding compute costs.

Methodology

1. Backbone encoder‑decoder: A pre‑trained multimodal LLM (vision + language) processes the chart image and produces a latent representation.
2. Mixture‑of‑Experts layer:
   • Each expert is a lightweight transformer block fine‑tuned for a subset of chart complexities (measured by element count, axis density, etc.).
   • A sparse gating network computes a routing score based on learned structural metrics, then selects the top‑k experts (k=1 in the final model) for each input.
   • Load‑balancing loss forces the gate to use all experts evenly, preventing “expert collapse.”
3. LoRA adapters: Instead of updating the whole backbone, low‑rank matrices (rank = 8) are injected into key linear layers. This yields a small set of trainable parameters while preserving the expressive power of the large model.
4. Training loop:
   • Phase 1: Freeze the backbone, train only LoRA adapters and the gating network.
   • Phase 2: Unfreeze a small subset of backbone layers together with LoRA, fine‑tuning the whole system end‑to‑end.
   • The loss combines code‑generation cross‑entropy with a routing‑stability term, encouraging consistent expert assignments across epochs.

Results & Findings

• Routing analysis shows that the gate learns intuitive splits (e.g., dense scatter plots go to Expert 3, simple pie charts to Expert 1).
• Ablation: Reducing experts to 4 drops accuracy by ~4 %; increasing rank beyond 8 yields diminishing returns while inflating memory.
• Scalability test: On a 2‑GPU setup, the model still fits comfortably, confirming suitability for production‑grade hardware.

Practical Implications

• Plug‑and‑play chart‑to‑code services: SaaS platforms can embed Chart2Code‑MoLA to auto‑generate D3.js, Matplotlib, or Vega‑Lite scripts from user‑uploaded images, dramatically reducing manual coding effort.
• Resource‑constrained environments: Because LoRA limits trainable parameters, the model can be fine‑tuned on a single GPU (or even on‑device with TensorRT‑style optimizations), making it viable for edge analytics dashboards.
• Modular extensibility: Adding a new chart family (e.g., Sankey diagrams) only requires training a dedicated expert rather than retraining the whole model, aligning with micro‑service architectures.
• Faster iteration cycles: The 20 % convergence speedup translates to shorter CI/CD pipelines for teams that continuously improve their visualization generators.
• Cross‑tool compatibility: Generated code can be targeted to multiple libraries by swapping the decoder head, allowing a single model to serve both web (JavaScript) and data‑science (Python) ecosystems.

Limitations & Future Work

• Domain coverage: The current experts are tuned on a fixed set of chart types; exotic or highly customized visualizations may still fall back to generic performance.
• Routing overhead: Although lightweight, the gating network adds a small latency penalty that could matter in ultra‑low‑latency applications.
• LoRA rank selection: The paper settles on rank‑8 empirically; a more systematic exploration (e.g., adaptive rank per expert) could yield further gains.
• Generalization beyond charts: Extending the MoE‑LoRA paradigm to other multimodal code‑generation tasks (e.g., UI mockup → HTML/CSS) is an open research direction.

Chart2Code‑MoLA demonstrates that smart modularity—routing inputs to specialized experts while keeping the bulk of the model frozen—can deliver both accuracy and efficiency. For developers building the next generation of automated visualization tools, the paper offers a concrete blueprint that balances performance with practical deployment constraints.

Authors

• Yifei Wang
• Jacky Keung
• Zhenyu Mao
• Jingyu Zhang
• Yuchen Cao

Paper Information

• arXiv ID: 2511.23321v1
• Categories: cs.SE
• Published: November 28, 2025
• PDF: Download PDF