[Paper] FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Source: arXiv - 2603.08708v1

Overview

The paper introduces Foreground View‑Guided Prompt Tuning (FVG‑PT), a lightweight add‑on that steers CLIP‑style Vision‑Language Models (VLMs) to keep their visual attention focused on the true foreground objects while they are being fine‑tuned with prompts. By detecting and correcting “foreground attention drift” that often causes prompt‑tuned models to misclassify, the authors achieve more reliable performance across a variety of downstream vision tasks.

Key Contributions

• Foreground Reliability Gate (FRG): a learnable module that evaluates the quality of the current foreground view and amplifies reliable foreground signals.
• Foreground Distillation Compensation (FDC): a distillation‑style loss that explicitly nudges the visual encoder’s attention maps toward foreground regions during prompt tuning.
• Prior Calibration (PC) Module: mitigates over‑focusing on foreground by re‑balancing the attention distribution with a calibrated prior, preserving generalization to background cues.
• Plug‑and‑play design: FVG‑PT can be attached to any CLIP‑based prompt‑tuning pipeline without retraining the underlying VLM.
• Extensive empirical validation: Demonstrates consistent gains on multiple backbones (ViT‑B/16, ViT‑L/14) and datasets (ImageNet‑R, CIFAR‑100, Oxford‑Pets, etc.), with code released for reproducibility.

Methodology

1. Detecting the problem: The authors first show that during prompt tuning, the visual encoder’s attention maps often shift away from the true foreground, leading to prediction errors.
2. Foreground Reliability Gate:
   • Takes the raw attention map from the visual encoder.
   • Passes it through a small MLP that learns a scalar “reliability” score per image.
   • Multiplies the original attention map by this score, boosting trustworthy foreground regions while suppressing noisy background noise.
3. Foreground Distillation Compensation:
   • Generates a pseudo‑ground‑truth foreground mask using off‑the‑shelf saliency detectors.
   • Applies a KL‑divergence loss between the gated attention distribution and the pseudo mask, encouraging the encoder to attend where the mask indicates foreground.
4. Prior Calibration:
   • Maintains a global prior attention distribution (learned across the training set).
   • Adds a regularization term that penalizes excessive deviation from this prior, preventing the model from becoming “myopic” to only foreground cues.
5. Training Loop: The three modules are trained jointly with the standard prompt‑tuning loss (cross‑entropy on the downstream task). Because they only touch the attention maps, the underlying VLM weights remain frozen.

Results & Findings

• Attention alignment: Visualizations show that after FVG‑PT the attention heatmaps tightly overlap with object silhouettes, whereas vanilla prompt tuning often lights up background textures.
• Robustness to domain shift: When evaluated on out‑of‑distribution variants (e.g., ImageNet‑A), the FVG‑PT‑enhanced models lose less accuracy, confirming that the prior calibration prevents over‑fitting to foreground only.
• Compatibility: Adding FVG‑PT to existing prompt‑tuning codebases incurs < 5 % extra FLOPs and < 2 M additional parameters, making it practical for real‑world pipelines.

Practical Implications

• Faster adaptation: Developers can keep the heavy VLM frozen and still gain a noticeable boost simply by plugging in FVG‑PT, saving GPU time and memory.
• Better reliability in safety‑critical apps: For tasks like autonomous inspection or medical image triage, ensuring the model looks at the right region is crucial; FVG‑PT provides an interpretable safeguard.
• Improved zero‑shot transfer: Since the method preserves a calibrated prior, models retain strong zero‑shot capabilities while being fine‑tuned for a specific domain.
• Ease of integration: The open‑source implementation works with Hugging Face’s transformers and open_clip libraries, so teams can adopt it with a few lines of code.
• Potential for multimodal extensions: The foreground‑guidance concept could be transferred to video‑language models (e.g., CLIP‑Video) or to text‑to‑image generation pipelines where foreground fidelity matters.

Limitations & Future Work

• Reliance on external saliency detectors: The pseudo‑foreground masks are generated by off‑the‑shelf saliency models, which may fail on highly cluttered or abstract images.
• Limited to vision‑centric tasks: The current formulation assumes a clear visual foreground; extending the idea to tasks where “foreground” is ambiguous (e.g., scene classification) remains open.
• Scalability to massive datasets: While the overhead is modest, training the reliability gate on millions of images could become a bottleneck; future work could explore lightweight self‑supervised gating.
• Exploration of joint vision‑language gating: The authors hint at a future direction where both visual and textual prompts are co‑adapted using a similar reliability signal.

The authors provide the full codebase at https://github.com/JREion/FVG-PT, making it straightforward for developers to experiment and integrate the technique into their own CLIP‑based solutions.

Authors

• Haoyang Li
• Liang Wang
• Siyu Zhou
• Jiacheng Sun
• Jing Jiang
• Chao Wang
• Guodong Long
• Yan Peng

Paper Information

• arXiv ID: 2603.08708v1
• Categories: cs.CV
• Published: March 9, 2026
• PDF: Download PDF