Source: Dev.to

Building & Integrating a Custom AI Voice Agent

Using the MirrorFly AI‑RAG Dashboard & SDK

The solution supports real‑time audio streaming, a visual workflow builder, and Retrieval‑Augmented Generation (RAG) for an external knowledge base. This lets you align the agent with your brand and deliver context‑aware user experiences.

---

Overview

We split the implementation into two parts:

---

Section I – 7 Steps to Build a Custom AI Voice Agent

1️⃣ Initial Setup

1. Get Access – Obtain developer credentials from the MirrorFly team and log in to the MirrorFly AI Dashboard.

2. Create an Agent – Click Create Agents → Voice Agent.

   

3. Configure the Agent – Enter a name, description, and an initial System Prompt that defines the core behavior.

   

---

2️⃣ Personality & Model Settings

• Personality – Set the welcome message, fallback responses, and adjust formality/tone.

• Model Selection – Choose from multiple LLM providers (no vendor lock‑in).

  

---

3️⃣ RAG Training

• Datasets – Upload PDFs or CSVs (max 5 MB per file, up to 10 files) for domain‑specific knowledge.

• Website Sync – Provide a URL to keep the agent’s knowledge up‑to‑date automatically.

  

---

4️⃣ Workflow Builder

Use the visual drag‑and‑drop canvas to define conversation logic:

• Conversational paths & decision trees
• API calls & form collection
• Email triggers & message nodes

---

5️⃣ Speech & Functions

• STT / TTS – Configure Speech‑to‑Text and Text‑to‑Speech providers (Deepgram, ElevenLabs, custom models, etc.).
• Call Handling – Enable end call, transfer to human (via SIP or conference), and set “interruption sensitivity”.
• Outbound Calling – Choose single‑call or batch‑call (CSV upload) modes and optionally enable post‑call analysis.

---

6️⃣ Custom Tools

• Webhook – Send real‑time event notifications to your backend for automation.

• MCP Server – Connect external services (Gmail, Google Calendar, Drive, etc.) to trigger actions like email sending, meeting scheduling, or file access.

  

---

7️⃣ Customize Widget

• Brand Personalization – Upload an avatar, set the agent name, choose primary theme colors, and style message bubbles.
• Live Preview – Test appearance and interaction flow before pushing to production.

---

Section II – Agent Integration

Prerequisites

• Valid Agent ID from the dashboard.
• Site served over HTTPS (required for microphone access).
• Supported browsers: Chrome, Edge, Safari (latest versions).

1️⃣ Install the SDK

Add the MirrorFly AI SDK to your HTML with a single script tag:

(Replace the URL with the production path when you go live.)

---

You’re now ready to embed the custom AI voice agent into your web application and start delivering intelligent, brand‑consistent voice interactions. 🚀

2. Initialize the Agent

Define a container element and initialize the SDK:

// HTML Container

// Initialization
MirrorFlyAi.init({
  container: "#widget",
  agentId: "",
  title: "Voice Assistant",
  theme: "dark",
  triggerStartCall: true,
  transcriptionEnable: true,
  transcriptionInUi: true,
  chatEnable: true,
  agentConnectionTimeout: 500
});

---

3. Handle Callbacks

Monitor the agent’s status and capture transcriptions:

const callbacks = {
  onTranscription: (data) => console.log("Transcription:", data),
  onAgentConnectionState: (state) => console.log("Connection:", state),
  onError: (error) => console.error("SDK Error:", error)
};

---

4. Dynamic Agent Switching

If your platform uses multiple agents (e.g., Sales vs. Support), use the following to switch contexts:

function switchAgent(newAgentId) {
  MirrorFlyAi.destroy();
  document.querySelector("#widget").innerHTML = "";
  MirrorFlyAi.init({
    container: "#widget",
    agentId: newAgentId,
    triggerStartCall: true
  });
}

---

5. Security & Permissions

The browser will prompt the user for microphone permission once the SDK is initialized.
Handle potential errors via the onError callback (e.g., permission denials or network issues).

---

Conclusion

In this guide, we demonstrated how to build, train, and deploy a custom AI Voice Agent using MirrorFly’s AI‑RAG dashboard and SDK.

• Full control over behavior, integrations, and deployment through RAG‑based knowledge integration, real‑time voice streaming, SDK embedding, and workflow orchestration.
• Extend capabilities with webhooks, MCP tools, and external APIs without modifying the core voice infrastructure.
• Adapt multiple business workflows (support, sales, internal automation).
• Support dynamic agent switching, transcription handling, and call‑control features.
• Ensure predictable performance across environments by properly configuring datasets, speech providers, and SDK callbacks.

Why wait? Start building your own AI voice agents on MirrorFly now. Use our GitHub custom AI voice agent repository and have your first agent running in hours.

Deploy a custom AI Voice Agent with MirrorFly’s AI‑RAG dashboard.