How to Prioritize Naturalness in Voice AI: Implement VAD

Source: Dev.to

TL;DR

Most voice AI breaks when users interrupt mid‑sentence or pause to think—the bot either talks over them or cuts them off. Voice Activity Detection (VAD) solves this by detecting speech boundaries in real‑time, enabling natural turn‑taking and barge‑in handling. Configure VAPI’s VAD thresholds, add back‑channel cues (e.g., “mm‑hmm”), and flush audio buffers on interruptions to avoid overlap. The result is conversations that feel human, not robotic.

API Access & Authentication

• VAPI API key – obtain from dashboard.vapi.ai
• Twilio Account SID and Auth Token – for phone number provisioning

Technical Requirements

• Public HTTPS endpoint for webhook handling (ngrok works for local development)
• Node.js 18+ with npm or yarn
• Basic knowledge of WebSocket connections and event‑driven architecture
• Familiarity with async/await in JavaScript

Voice AI Fundamentals

• VAD thresholds and their impact on latency
• Turn‑taking mechanics (detecting when the user stops speaking)
• Barge‑in behavior (interrupting the bot mid‑sentence)
• Real‑time audio streaming constraints (16 kHz PCM, μ‑law encoding)

Production Considerations

• Budget: ~ $0.02–$0.05 per minute for combined STT + TTS

Audio Processing Pipeline

graph TD
    AudioCapture[Audio Capture] --> VAD[Voice Activity Detection]
    VAD --> STT[Speech‑to‑Text]
    STT --> LLM[Large Language Model]
    LLM --> TTS[Text‑to‑Speech]
    TTS --> AudioOutput[Audio Output]

    STT -->|Error| ErrorHandling[Error Handling]
    LLM -->|Error| ErrorHandling
    TTS -->|Error| ErrorHandling
    ErrorHandling -->|Retry| AudioCapture

The pipeline processes audio in 20 ms frames:

1. User speaks → audio buffered in 20 ms frames
2. VAD analyzes energy levels
3. If silence ≥ endpointing duration → flush buffer to STT
4. Transcript sent to LLM → response synthesized → streamed back

A race condition can occur when VAD fires while STT is still processing the previous chunk, leading to duplicate responses. Guard against this with explicit turn‑state tracking.

Step 1: Configure Twilio for Inbound Calls

// Your server receives Twilio webhook
app.post('/voice/inbound', async (req, res) => {
  const twiml = `
    <?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Gather input="speech" action="/voice/handle" method="POST">
        <Say>Welcome, please tell me how I can help.</Say>
      </Gather>
    </Response>
  `;
  res.type('text/xml');
  res.send(twiml);
});

Step 2: Implement Backchanneling via Prompt Engineering

const systemPrompt = `You are a natural conversationalist. Rules:
1. Use backchannels ("mm-hmm", "I see", "go on") when user pauses mid‑thought.
2. Detect incomplete sentences (trailing "and...", "so...") and wait.
3. Keep responses under 15 words unless the user asks for detail.
4. Never say "How can I help you?" – jump straight to the topic.`;

Backchannels are generated by the LLM, not by VAD.

Step 3: Handle Barge‑in at the Audio Buffer Level

const callConfig = {
  assistant: assistantConfig,
  backgroundSound: "office", // Enables barge‑in detection
  recordingEnabled: true
};

When VAD detects new speech during TTS playback, the audio buffer must be flushed immediately. VAPI does this automatically when backgroundSound is set.

Testing Guidelines

• Pause test: Call your Twilio number, speak a sentence, pause 300 ms, then continue. The bot should not interrupt. If it does, increase endpointing by 50 ms increments.
• Barge‑in test: Start speaking while the bot is talking. Audio should cut within ~200 ms. Verify backgroundSound is enabled.
• Noise robustness: Test in noisy environments (coffee shop, car). If false positives occur, raise the endpointing to 300 ms+.

Example VAD Threshold Test

const testVADConfig = {
  transcriber: {
    provider: "deepgram",
    model: "nova-2",
    language: "en",
    endpointing: 200 // aggressive start
  }
};

async function testBargeIn() {
  const startTime = Date.now();
  console.log('Testing barge‑in at 1.2 s into TTS playback...');
  if (Date.now() - startTime > 300) {
    console.error('VAD latency exceeded 300 ms – adjust endpointing');
  }
}
testBargeIn();

Webhook Signature Verification

// Verify VAPI webhook signature
app.post('/webhook/vapi', (req, res) => {
  const signature = req.headers['x-vapi-signature'];
  const secret = process.env.VAPI_SECRET;
  // (Insert HMAC verification logic here)
  if (!isValidSignature(signature, secret, req.body)) {
    return res.status(401).send('Invalid signature');
  }
  // Process webhook payload...
  res.sendStatus(200);
});