[Paper] Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models

Source: arXiv - 2603.04292v1

Overview

The paper argues that the next step in making large language models (LLMs) production‑ready is to expose vector‑prompt interfaces—continuous embeddings that can be fed to the model at inference time—rather than relying solely on textual prompts. The authors show that vector prompts scale better with supervision, exhibit richer attention patterns, and can be used for stable, inference‑only customization without increasing security risks.

Key Contributions

• Position statement: Proposes that LLM providers should make vector‑prompt inputs part of the public API for model customization.
• Empirical evidence: Demonstrates that vector‑prompt tuning continues to improve as more labeled data are added, while text‑only prompt optimization plateaus early.
• Attention analysis: Shows that vector prompts trigger dense, global attention across the model, suggesting a fundamentally different control mechanism from token‑level prompts.
• Security assessment: Argues that exposing vector prompts does not materially raise model‑leakage risk under a standard black‑box threat model.
• Call to action: Encourages the community to treat prompt interfaces as a first‑class, configurable component of LLM services.

Methodology

1. Prompt Types Compared

   • Textual prompts: Hand‑crafted or automatically optimized strings inserted into the input.
   • Vector prompts: Learned continuous embeddings (e.g., a small set of trainable vectors) concatenated to the model’s hidden states before the first transformer layer.

2. Training Regime

   • Both prompt types were fine‑tuned on a suite of downstream tasks (classification, QA, summarization) with varying amounts of labeled data (from 0.1 % to 100 % of the full training set).
   • Optimization used standard gradient descent on the prompt parameters only; the underlying LLM weights remained frozen (inference‑only customization).

3. Evaluation Metrics

   • Task performance (accuracy, F1, ROUGE, etc.).
   • Saturation curves to see how performance scales with supervision.
   • Attention heatmaps to visualize how prompts influence token‑level attention.

4. Security Analysis

   • Simulated black‑box attacks (prompt‑injection, model extraction) to measure any increase in leakage when vector prompts are exposed.

Results & Findings

Takeaway: Vector prompts provide a more expressive and scalable knob for customizing LLM behavior, while remaining safe to expose.

Practical Implications

• Product teams can ship “plug‑and‑play” customization modules (e.g., domain‑specific adapters) without retraining the entire model, reducing compute costs and time‑to‑market.
• Developers gain a deterministic API: send a small set of vectors (often < 1 KB) alongside the user query to tailor tone, style, or factual grounding on the fly.
• MLOps pipelines can version‑control vector prompts just like model weights, enabling A/B testing and rollback without touching the base LLM.
• Compliance & governance: Since the base model stays frozen, audit logs can focus on prompt changes, simplifying traceability for regulated industries.
• Marketplace ecosystems: Third‑party vendors could sell “prompt bundles” (e.g., legal‑ese, medical jargon) that are interoperable across any provider exposing the vector‑prompt endpoint.

Limitations & Future Work

• Hardware overhead: Concatenating vectors adds a modest memory and compute cost, which may be non‑trivial for extremely latency‑sensitive services.
• Prompt size selection: The optimal number of vectors varies per task; the paper does not provide a universal recipe.
• Black‑box threat model: The security analysis assumes attackers cannot observe internal activations; stronger white‑box or side‑channel attacks remain unexplored.
• Generalization to multimodal models: Extending vector prompts to vision‑language or audio models is an open question.

Future directions include automated methods for sizing vector prompts, benchmarking on ultra‑large LLMs (≥ 100 B parameters), and exploring hybrid interfaces that combine textual and vector cues for even richer control.

Authors

• Liangwei Yang
• Shiyu Wang
• Haolin Chen
• Rithesh Murthy
• Ming Zhu
• Jielin Qiu
• Zixiang Chen
• Juntao Tan
• Jianguo Zhang
• Zhiwei Liu
• Wenting Zhao
• Silvio Savarese
• Caiming Xiong
• Huan Wang
• Shelby Heinecke

Paper Information

• arXiv ID: 2603.04292v1
• Categories: cs.CL
• Published: March 4, 2026
• PDF: Download PDF