[Paper] Self-Prophetic Decoding to Unlock Visual Search in LVLMs

Source: arXiv - 2605.28741v1

Overview

Large Vision‑Language Models (LVLMs) are getting better at “thinking with images,” but when they are asked to perform visual search—locating objects or concepts across a scene—they still stumble. This paper introduces Self‑Prophetic Decoding (SeProD), a training‑free, plug‑and‑play technique that lets a post‑trained LVLM borrow the reliable single‑step reasoning abilities of its pre‑training ancestor, dramatically improving multi‑step visual search without extra compute.

Key Contributions

• Self‑regulation insight: Demonstrates that the pre‑training model’s single‑step competence can be used to counteract capability loss and long‑context interference that arise after fine‑tuning.
• Probabilistic prophetic sampling: Replaces naïve prompting with a probability‑based token “prophecy” mechanism, where the pre‑training model predicts useful tokens and the post‑training model selectively adopts them.
• SeProD framework: A lightweight decoding strategy that works with any existing LVLM, requires no additional training, and runs in parallel to keep latency unchanged.
• Comprehensive evaluation: Shows consistent gains across four visual‑search benchmarks (12 splits total) and several general VQA datasets, proving the method’s generality.

Methodology

1. Dual‑model setup – Keep both the original pre‑training LVLM (the “prophet”) and the fine‑tuned LVLM (the “executor”) alive during inference.
2. Prophetic token generation – For each decoding step, the prophet samples a set of candidate tokens from its probability distribution (instead of a single deterministic token).
3. Selective acceptance – The executor examines the same step’s probability distribution and accepts only those prophetic tokens that it deems plausible (i.e., they have non‑negligible probability under the executor).
4. Parallel decoding – Both models run side‑by‑side, so the extra sampling does not increase wall‑clock time; the executor simply merges the accepted prophetic tokens into its own output stream.
5. Training‑free integration – Because the process only touches the decoding stage, any LVLM that already supports standard auto‑regressive generation can adopt SeProD without retraining or architectural changes.

Results & Findings

• Visual search benchmarks: SeProD lifts accuracy by 3–7 % across all 12 splits of the four datasets, closing gaps that fine‑tuning alone could not.
• General VQA tasks: Gains of 1.5–2.8 % on standard VQA benchmarks indicate that the benefits extend beyond visual search.
• No extra latency: The parallel prophetic acceptance adds negligible overhead, preserving real‑time inference speeds.
• Robustness to long contexts: The method mitigates the “drift” that typically occurs when LVLMs handle multi‑turn dialogs or lengthy reasoning chains.

Practical Implications

• Plug‑and‑play upgrade: Developers can boost existing LVLM‑powered products (e.g., image‑based assistants, visual QA bots, AR search tools) simply by swapping in the SeProD decoder.
• Cost‑effective performance: Since no additional training or larger models are required, teams can achieve higher accuracy without extra GPU budgets.
• Improved user experience: More reliable multi‑step visual reasoning translates to fewer misunderstandings in applications like visual troubleshooting, e‑commerce visual search, and interactive robotics.
• Framework‑agnostic: The approach works with any transformer‑based LVLM that exposes token probabilities, making it compatible with open‑source models (e.g., BLIP‑2, LLaVA) and proprietary APIs.

Limitations & Future Work

• Dependency on a strong pre‑training model: If the original pre‑training LVLM is weak, the prophetic tokens may not provide useful guidance.
• Heuristic acceptance threshold: The current selection rule is simple (probability‑based); more sophisticated criteria (e.g., confidence calibration or learned gating) could further improve performance.
• Scope of tasks: While visual search and VQA benefit, the paper does not explore other multimodal tasks such as captioning or video reasoning—future work could test SeProD’s versatility there.
• Theoretical analysis: A deeper formal understanding of why prophetic sampling stabilizes long‑context reasoning remains an open research question.

SeProD shows that a clever decoding tweak can unlock latent capabilities in LVLMs, offering a practical shortcut for developers eager to deliver smarter visual‑search experiences today.

Authors

• Zhendong He
• Qiyuan Dai
• Guanbin Li
• Liang Lin
• Sibei Yang

Paper Information

• arXiv ID: 2605.28741v1
• Categories: cs.CV
• Published: May 27, 2026
• PDF: Download PDF