[Paper] FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models

Source: arXiv - 2512.20561v1

Overview

FlashVLM tackles a core bottleneck in large vision‑language models (VLMs): the massive number of visual tokens that must be processed for every image or video frame. By selecting only the tokens that are truly relevant to a given text query, FlashVLM slashes the quadratic attention cost while actually improving performance on many benchmarks. The paper shows that you can prune up to ≈ 78 % of visual tokens without sacrificing accuracy, and even exceed the unpruned baseline in some cases.

Key Contributions

• Text‑guided token selection: Introduces an explicit cross‑modal similarity score between image patches and the query embedding, rather than relying on noisy self‑attention maps.
• Hybrid relevance weighting: Combines extrinsic (text‑query relevance) and intrinsic (visual saliency) cues using log‑domain weighting and temperature‑controlled sharpening for robust ranking.
• Diversity‑preserving partition: Guarantees a minimal set of background tokens to retain global scene context, preventing over‑pruning of “boring” regions.
• Lossless‑or‑better compression: Demonstrates that, at identical token budgets, FlashVLM can match or surpass the original model’s accuracy (e.g., 92.8 % of original performance with 94.4 % compression).
• Broad evaluation: Validated on 14 image and video datasets across multiple VLM backbones (including LLaVA‑1.5), showing consistent efficiency‑accuracy trade‑offs and strong robustness.

Methodology

1. Project visual tokens: Each image patch (or video frame token) is linearly projected into the same embedding space used by the language model.
2. Compute cross‑modal similarity: The projected token is dotted with the normalized text‑query embedding, yielding a relevance score that directly measures how “talk‑about‑able” a patch is for the given prompt.
3. Fuse with visual saliency: An intrinsic saliency map (derived from a lightweight CNN or the VLM’s early layers) is combined with the relevance score. The fusion happens in the log domain and is sharpened by a temperature parameter, which accentuates high‑relevance tokens while suppressing noise.
4. Rank & prune: Tokens are sorted by the fused score. A user‑defined budget (e.g., keep 20 % of tokens) determines the cutoff.
5. Diversity partition: To avoid discarding all background information, FlashVLM reserves a small quota of low‑scoring tokens that are spatially diverse, preserving a coarse global context.
6. Feed pruned set to the VLM: The reduced token set is passed through the standard transformer layers, incurring far less quadratic attention cost.

The whole pipeline is lightweight (no extra deep attention passes) and can be plugged into any existing VLM that already exposes token embeddings.

Results & Findings

• State‑of‑the‑art efficiency: Across all tested VLMs (LLaVA‑1.5, MiniGPT‑4, etc.), FlashVLM consistently outperformed prior token‑reduction methods (e.g., attention‑based pruning, uniform down‑sampling).
• Robustness: Even under extreme compression (≥ 94 % token removal), performance degradation was graceful, and the model retained strong zero‑shot capabilities on out‑of‑distribution prompts.
• Generalization: The same relevance‑fusion hyper‑parameters transferred across image and video tasks, indicating that the approach is not tightly coupled to a specific dataset.

Practical Implications

• Cost‑effective inference: Deploying VLMs on edge devices, mobile GPUs, or serverless environments becomes feasible because the quadratic attention cost is dramatically reduced.
• Faster interactive AI assistants: Real‑time multimodal chatbots (e.g., LLaVA‑based agents) can respond faster, enabling smoother user experiences in AR/VR or web‑based applications.
• Scalable video analytics: Processing every frame of a video is traditionally prohibitive; FlashVLM’s token selection can be applied frame‑wise, cutting compute by an order of magnitude while preserving the ability to answer frame‑specific questions.
• Energy savings: Lower FLOPs translate directly into reduced power consumption—an attractive proposition for large‑scale inference farms and sustainability‑focused deployments.
• Plug‑and‑play: Since the method works on top of existing token embeddings, developers can integrate FlashVLM into their pipelines with minimal code changes (e.g., a preprocessing hook before the transformer encoder).

Limitations & Future Work

• Dependency on a good text embedding: If the language model’s query representation is weak (e.g., ambiguous prompts), relevance scores may mislead the pruning decision.
• Static budget selection: The current implementation uses a fixed token budget per image; adaptive budgets based on scene complexity could yield even better trade‑offs.
• Limited to transformer‑style VLMs: Models that do not expose token‑level embeddings (e.g., some diffusion‑based multimodal systems) would need additional engineering.
• Future directions: The authors suggest exploring learned temperature schedules, incorporating multimodal feedback loops (e.g., re‑ranking after a first pass), and extending the framework to 3‑D point‑cloud or LiDAR data for autonomous‑driving scenarios.

Authors

• Kaitong Cai
• Jusheng Zhang
• Jing Yang
• Yijia Fan
• Pengtao Xie
• Jian Wang
• Keze Wang

Paper Information

• arXiv ID: 2512.20561v1
• Categories: cs.CV
• Published: December 23, 2025
• PDF: Download PDF