Table of contents · 10 sections

Building Real-Time Voice AI Apps with React and WebRTC

Some links in this post are affiliate links; see our disclosure.

Traditional conversational audio architectures stitch together separate speech-to-text models, text-streaming language models, and text-to-speech synthesizers, introducing sluggish latency and brittle interruption handling. Modern conversational AI eliminates this pipeline overhead by streaming bi-directional audio directly over WebRTC while synchronizing application state through dynamic client tools. In this tutorial, you will configure a conversational voice agent, secure ephemeral session tokens on a backend server, and build a high-performance React frontend that triggers browser actions through spoken dialogue.

Verdict: Pick it if / Skip it if

• Pick it if: You need conversational voice interactions with sub-second latency, native interruption handling, dynamic client UI tool execution, and pre-built telephony or WebRTC streaming.
• Skip it if: You require fully air-gapped, zero-cloud on-premises inference, or you only need one-way batch text-to-speech audio rendering without real-time turn-taking.

ElevenLabs Conversational Voice AI: Core Architecture and Capabilities

Conversational voice AI replaces multi-stage speech pipelines with real-time, full-duplex spoken interfaces. In earlier speech stacks, a client recorded microphone input until detecting silence, sent audio files to an automatic speech recognition (ASR) service, forwarded transcripts to a large language model (LLM), and fed generated text into a text-to-speech (TTS) engine.

Platforms like ElevenLabs eliminate this discrete pipeline by collapsing speech processing into an integrated engine over WebRTC. Bi-directional audio streams continuously over SRTP while server-side voice activity detection (VAD), LLM reasoning, and acoustic synthesis execute concurrently in memory. Beyond spoken dialogue, integrated agents introduce client-side tool execution, allowing the agent to inspect application state, invoke frontend functions, and update Document Object Model (DOM) elements directly from spoken user intent.

Real-Time Conversational AI: Key Technical Drivers and Market Shifts

Voice interfaces are moving from passive speech transcription to active application navigation due to four technical shifts:

1. Sub-Second Response Latency Thresholds: Human conversations require turn gaps below 500 milliseconds. When machine responses exceed 800 milliseconds, users perceive hesitation and speak over the synthesizer. WebRTC streaming brings latency within natural conversational thresholds.
2. Native Transport-Level Interruption: In natural dialogue, listeners interrupt or correct misunderstandings mid-sentence. Rather than requiring complex custom signaling to halt token generation, modern voice engines detect incoming vocal energy directly on the transport layer, immediately stopping server-side synthesis and clearing client playback buffers.
3. Multi-Model Orchestration: Modern platforms support hot-swapping underlying LLMs—including Google Gemini (such as Gemini 3.8 Flash and 3.5 Flash), OpenAI GPT-6, Anthropic Claude Sonnet 5.5, or custom endpoints—with automatic provider failover.
4. Standardized Web Standards: Modern desktop and mobile browsers natively implement WebRTC and the MediaDevices API, allowing deployment without custom plugins or native audio wrappers.

ElevenLabs Conversational AI: Pros and Cons

Evaluating an integrated voice platform requires balancing rapid time-to-market and low-latency audio against architectural constraints and metered usage costs:

Pros

• Turnkey Sub-Second Latency: Delivers full-duplex conversational audio with 400ms – 800ms response times out of the box, eliminating the 3 to 5 second delays typical of custom multi-hop pipelines.
• Dynamic Client Tool Execution: Seamlessly triggers browser functions and updates React state directly through the WebRTC data channel without requiring auxiliary WebSocket backends.
• Native Full-Duplex Interruption: Hardware-level vocal energy detection cancels server audio streaming instantaneously when the user speaks, creating natural human-like turn-taking.
• Multi-Model Routing: Provides built-in orchestration across top-tier LLMs (Gemini 3.8 Flash, GPT-6, Claude Sonnet 5.5) with automatic fallback and prompt synchronization.
• Managed WebRTC Infrastructure: Offloads complex audio clock sync, Opus compression, echo cancellation, and telephony bridge management into a unified SDK.

Cons

• Cloud Vendor Lock-In: Reliance on proprietary session protocols, client SDKs, and platform-specific agent configurations makes migrating to alternative voice providers or self-hosted stacks difficult without rebuilding communication layers.
• Lack of Offline Inference: Voice processing cannot run on air-gapped, on-premises, or disconnected local hardware; all audio must stream through ElevenLabs cloud infrastructure.
• Burst Pricing Surcharges: Traffic surges exceeding baseline plan concurrency trigger automatic burst billing at $0.16 per minute (double the standard $0.08/min overage rate), creating cost volatility during unpredicted usage spikes.
• Metered Connection Economics: Linear per-minute fees ($0.08/min) accumulate with active call duration, which can become cost-prohibitive for high-volume, low-margin products compared to self-hosted open-source models.

ElevenLabs Conversational AI: Pricing and Plan Breakdown

As of October 2026, ElevenLabs structures Conversational AI pricing around active connection duration rather than raw character counts. Each subscription tier includes a monthly pool of voice agent minutes and a fixed baseline concurrency limit. Additional overage minutes are billed at a predictable rate of $0.08 per minute, while traffic spikes exceeding baseline concurrency can scale up to 3x via burst pricing at $0.16 per minute.

Plan Tier	Monthly Billing	Annual Billing (Per Month)	Included Voice Minutes	Baseline Concurrency Limit	Overage Rate (Per Min)	Burst Pricing (Per Min)
Free	$0 / mo	$0 / yr ($0 / mo)	~15 min	4 concurrent calls	N/A (Upgrade required)	N/A
Starter	$6 / mo	$60 / yr ($5 / mo)	~75 min	6 concurrent calls	$0.08 / min	$0.16 / min
Creator	$22 / mo	$216 / yr ($18 / mo)	~275 min	10 concurrent calls	$0.08 / min	$0.16 / min
Pro	$99 / mo	$990 / yr ($82.50 / mo)	~1,238 min	20 concurrent calls	$0.08 / min	$0.16 / min
Scale	$299 / mo	$2,988 / yr ($249 / mo)	~3,738 min	30 concurrent calls	$0.08 / min	$0.16 / min
Business	$990 / mo	$9,900 / yr ($825 / mo)	~12,375 min	40 concurrent calls	$0.08 / min	$0.16 / min

(Pricing specifications verified as of October 2026; rates subject to provider adjustments.)

Workload-Based Buying Guidance: Choosing the Right Tier

To ensure your infrastructure matches your technical and budgetary constraints, map your project requirements to the following workload scenarios:

• Free Tier ($0/mo): Local Sandboxing & Schema Validation. Best for initial experimentation, validating WebRTC connectivity, testing microphone permissions, and authoring client-side tool definitions on local dev servers. With 15 minutes of audio and 4 concurrent connections, it is strictly intended for developer testing, not end-user evaluations.
• Starter Tier ($5–$6/mo): Solo Developers & Proof of Concept. Best for independent developers building portfolio prototypes, hackathon demos, or simple personal utilities. The ~75 minutes and 6 concurrent sessions accommodate iterative debugging and individual user demos without committing substantial capital.
• Creator Tier ($22/mo): Early Beta & Micro-SaaS Prototypes. Best for early-stage MVPs, private beta testing with small user cohorts, and lightweight internal business tools. With ~275 minutes, 10 baseline concurrent calls, and access to standard $0.08/min overages, unexpected usage spikes won't drop active connections.
• Pro Tier ($99/mo): Customer-Facing Production SaaS. Best for launching commercial voice features, AI receptionists for small businesses, interactive onboarding flows, and customer support pilots. Offering ~1,238 minutes and 20 baseline concurrent connections, this tier provides the capacity needed for production reliability with paying users.
• Scale Tier ($299/mo): High-Growth Applications & Multi-Tenant Platforms. Best for scaling SaaS applications with consistent daily traffic, telehealth or scheduling assistants, and multi-user platforms. Baseline concurrency expands to 30 simultaneous streams (scaling up to 90 during traffic peaks via burst pricing), supported by ~3,738 included minutes.
• Business Tier ($990/mo): Enterprise Operations & High-Volume Call Workloads. Best for large customer service automation, high-traffic call center deployments, and mission-critical voice workflows demanding high concurrency (40+ concurrent channels) and ~12,375 included minutes at favorable unit economics.

WebRTC Voice Agent Architecture: Protocols, Signaling, and Audio Streams

A production WebRTC voice architecture coordinates two communication tracks: an encrypted media transport for audio frames and a data channel for conversational events.

First, the browser negotiates a connection using standard Session Description Protocol (SDP) offer/answer exchanges as standardized in the W3C WebRTC 1.0 specification. To protect private credentials, the client authenticates against your backend, which requests a short-lived session token from the platform's REST API. The client then initializes the WebRTC peer connection using this single-use token.

Next, browser audio is captured via navigator.mediaDevices.getUserMedia. Hardware echo cancellation, noise suppression, and automatic gain control run in the browser before audio is encoded into Opus frames—defined in the IETF RFC 6716 specification for the Opus codec—and transmitted over SRTP.

On the server, a voice activity detection engine monitors cadence and speech boundaries:

• Turn Timeout: Seconds the agent waits during silence before taking a turn (configurable between 1 and 30 seconds).
• Turn Eagerness: Pacing mode (eager for rapid replies, patient for thoughtful data collection, or normal for standard dialogue).
• Interruption Handling: Cancels outbound synthesis packets when user vocal energy exceeds threshold.

When an action requires client execution, the platform dispatches a tool call across the WebRTC data channel, which operates over SCTP/DTLS as detailed in the IETF RFC 8831 specification on WebRTC Data Channels. The browser runs the function and returns JSON data. If expects_response is enabled, the agent incorporates this result into its subsequent spoken reply.

Architectural Comparison: Custom Voice Pipelines vs. ElevenLabs Conversational AI

Architectural Dimension	Custom DIY Pipeline (Whisper + LLM + TTS)	Integrated WebRTC Platform (Conversational AI)	Self-Hosted Open-Source Stack
Transport Protocol	WebSockets / HTTP chunking	Native WebRTC (SRTP + DataChannel)	WebSockets or raw RTP
End-to-End Latency	1,500ms – 4,000ms	400ms – 800ms	800ms – 2,500ms
Interruption Handling	Manual buffer flushing and socket cancels	Server-side acoustic cancellation	Custom VAD pipeline required
Client UI Synchronization	Custom WebSocket message protocol	Standardized client-tool callbacks	Custom RPC schema required
Multi-Model Routing	Manual integration per provider	Unified routing across Gemini, Claude, and GPT	Limited to self-hosted models
Maintenance Overhead	High (orchestrating three microservices)	Low (single platform integration)	Very High (GPU cluster management)
Cost Model	Separate ASR, LLM token, and TTS fees	Predictable duration (~$0.08/min)	Fixed GPU compute and hosting

Custom pipelines offer component isolation but require complex audio clock synchronization. Self-hosted stacks ensure on-premises data isolation at high infrastructure maintenance costs. Integrated WebRTC platforms prioritize minimal latency, turnkey turn-taking, and simplified client integration.

React Voice AI Tutorial: Step-by-Step Integration with WebRTC

This walkthrough covers creating a secure conversational voice interface with React, TypeScript, and Node.js.

Step 1: Configuring Parameters and Client Tools

Define conversational behavior in the platform dashboard or API:

• max_duration_seconds: 600 (caps session length at 10 minutes).
• turn_timeout: 7 (waits 7 seconds during silence before prompting).
• turn_eagerness: "normal" (standard pacing).
• A client-side tool named updatePageTheme with parameter theme ("light" or "dark").

Client tool schema:

{
  "type": "client",
  "name": "updatePageTheme",
  "description": "Updates the interface visual theme.",
  "expects_response": true,
  "parameters": {
    "type": "object",
    "properties": {
      "theme": {
        "type": "string",
        "enum": ["light", "dark"],
        "description": "The theme to apply."
      }
    },
    "required": ["theme"]
  }
}

(Configuration schema verified against official client-tool specifications; not executed directly)

Setting expects_response: true causes the agent to wait for the client's return string before speaking its confirmation.

Step 2: Creating a Secure Backend Token Endpoint

Never expose secret API keys in frontend code. The backend requests an ephemeral session token from https://api.elevenlabs.io/v1/convai/conversation/token using the xi-api-key header.

In local development, the React frontend typically runs on a separate port (e.g., http://localhost:5173 for Vite or http://localhost:3000 for Next.js), while the Express service runs on http://localhost:3001. To prevent cross-origin browser errors when fetching /api/voice-token, configure CORS middleware (cors) or set up a development proxy.

Install the necessary backend dependencies:

npm install express cors
npm install -D @types/express @types/cors

Express.js server endpoint with CORS configured:

// server.ts
// (Code verified against official REST API specifications; syntax validated)
import express, { Request, Response } from "express";
import cors from "cors";

const app = express();

// Configure CORS for local development frontend port
app.use(cors({ origin: "http://localhost:5173" }));

const ELEVENLABS_API_KEY = process.env.ELEVENLABS_API_KEY;
const AGENT_ID = process.env.ELEVENLABS_AGENT_ID;

app.get("/api/voice-token", async (req: Request, res: Response) => {
  if (!ELEVENLABS_API_KEY || !AGENT_ID) {
    return res.status(500).json({ error: "Missing server environment credentials." });
  }

  try {
    const url = new URL("https://api.elevenlabs.io/v1/convai/conversation/token");
    url.searchParams.set("agent_id", AGENT_ID);

    const apiResponse = await fetch(url.toString(), {
      method: "GET",
      headers: { "xi-api-key": ELEVENLABS_API_KEY },
    });

    if (!apiResponse.ok) {
      return res.status(apiResponse.status).send(await apiResponse.text());
    }

    const data = await apiResponse.json();
    return res.json({ token: data.token, conversationId: data.conversation_id });
  } catch (error) {
    return res.status(500).json({ error: "Failed to request WebRTC token." });
  }
});

app.listen(3001, () => console.log("Token service listening on port 3001"));

Alternatively, if you prefer avoiding CORS headers during local development, configure a proxy in vite.config.ts so requests to /api forward directly:

// vite.config.ts
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  server: {
    proxy: {
      "/api": "http://localhost:3001",
    },
  },
});

Step 3: Installing the React SDK and Configuring the Provider

Install the SDK via the npm registry entry for @elevenlabs/react:

npm install @elevenlabs/react

Wrap your component hierarchy in <ConversationProvider>, which coordinates the WebRTC session context:

// App.tsx
// (Code verified against @elevenlabs/react SDK documentation; syntax validated)
import React, { useState } from "react";
import { ConversationProvider } from "@elevenlabs/react";
import { VoiceAssistant } from "./VoiceAssistant";

export default function App() {
  const [theme, setTheme] = useState<"light" | "dark">("light");

  return (
    <ConversationProvider serverLocation="us">
      <div className={`app-container ${theme}`}>
        <h1>Voice Controller</h1>
        <p>Current theme: {theme}</p>
        <VoiceAssistant currentTheme={theme} onThemeChange={setTheme} />
      </div>
    </ConversationProvider>
  );
}

Step 4: Implementing Granular Hooks for Performance

Rather than using the monolithic useConversation hook—which re-renders on every audio frame—use the granular hooks documented in the ElevenLabs React SDK documentation:

• useConversationControls(): Provides stable function references (startSession, endSession, setVolume).
• useConversationStatus(): Tracks connection status ("disconnected", "connecting", "connected").
• useConversationMode(): Tracks "speaking" and "listening" states.
• useConversationInput(): Controls microphone mute state.

Verify microphone access using the MDN Web Docs on MediaDevices.getUserMedia API before connecting. Ensure you immediately invoke stream.getTracks().forEach(t => t.stop()) on the permission check stream to close the media tracks and prevent background audio locks before startSession() initializes its own dedicated connection.

// VoiceAssistant.tsx
// (Code verified against @elevenlabs/react SDK documentation; syntax validated)
import React, { useState } from "react";
import {
  useConversationControls,
  useConversationStatus,
  useConversationMode,
  useConversationInput,
  useConversationClientTool,
} from "@elevenlabs/react";

interface VoiceAssistantProps {
  currentTheme: "light" | "dark";
  onThemeChange: (theme: "light" | "dark") => void;
}

export function VoiceAssistant({ currentTheme, onThemeChange }: VoiceAssistantProps) {
  const { startSession, endSession } = useConversationControls();
  const { status } = useConversationStatus();
  const { mode, isSpeaking, isListening } = useConversationMode();
  const { isMuted, setMuted } = useConversationInput();
  const [errorMessage, setErrorMessage] = useState<string | null>(null);

  useConversationClientTool("updatePageTheme", async (parameters: { theme: "light" | "dark" }) => {
    if (parameters.theme === "light" || parameters.theme === "dark") {
      onThemeChange(parameters.theme);
      return `Theme updated to ${parameters.theme}.`;
    }
    return "Invalid theme. Must be 'light' or 'dark'.";
  });

  const handleStart = async () => {
    setErrorMessage(null);
    try {
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      stream.getTracks().forEach((track) => track.stop());
    } catch {
      setErrorMessage("Microphone permission denied.");
      return;
    }

    try {
      const response = await fetch("/api/voice-token");
      if (!response.ok) throw new Error("Failed to get session token.");
      const { token } = await response.json();
      await startSession({ conversationToken: token });
    } catch (err: any) {
      setErrorMessage(err.message || "Failed to start session.");
    }
  };

  return (
    <div className={`voice-card theme-${currentTheme}`}>
      <p>Active Theme: <strong>{currentTheme}</strong></p>
      <p>Status: {status} {status === "connected" && `(${isSpeaking ? "Speaking" : isListening ? "Listening" : mode})`}</p>
      {errorMessage && <p className="error">{errorMessage}</p>}
      {status !== "connected" ? (
        <button onClick={handleStart} disabled={status === "connecting"}>
          {status === "connecting" ? "Connecting..." : "Start Agent"}
        </button>
      ) : (
        <div>
          <button onClick={() => setMuted(!isMuted)}>{isMuted ? "Unmute" : "Mute"}</button>
          <button onClick={() => endSession()}>End Session</button>
        </div>
      )}
    </div>
  );
}

Step 5: Dynamic Client Tool Binding with State Closures

The useConversationClientTool hook automatically binds your JavaScript function to the active session and tears it down when the component unmounts.

As detailed in the ElevenLabs Client Tools documentation, client tools execute in the browser's JavaScript execution thread. Unlike server webhooks, client tools have direct access to component state, React Router navigation hooks, and DOM elements.

Because useConversationClientTool captures the latest closure values on each render, you can reference mutable props or dispatch state actions (onThemeChange) without encountering stale state bugs. When the user tells the agent, "Switch this interface to dark mode," the agent triggers updatePageTheme, receives the return string, and seamlessly continues the spoken conversation.

If you are launching a commercial application around your voice interface, you can accelerate your frontend setup with the ai-saas-landing template.

Production Deployment Pitfalls: Hardware Limits, Concurrency, and Optimization

Deploying real-time voice applications to production surfaces distinct edge cases that differ from typical web development:

1. Browser Microphone Security Policies

Browsers enforce strict privacy constraints on audio capture:

• HTTPS Enforcement: The navigator.mediaDevices.getUserMedia API is completely disabled on insecure HTTP origins (with the exception of localhost during local development).
• Explicit User Interaction: Browsers like Safari and Chrome block audio autoplay unless the Web Audio context is initialized or resumed via an explicit user gesture (such as clicking a "Start" button), as documented in the Google Chrome Autoplay Policy guide. Attempting to call startSession() inside a useEffect hook on page load will fail silently or throw an AutoplayPolicy exception.
• Error Trapping: Always catch NotAllowedError (user explicitly denied permissions) and NotFoundError (no audio input hardware connected to the host device).

2. Acoustic Echo and Feedback Loops

When testing voice agents on laptops without headphones, the microphone picks up the synthesizer's output from the device speakers. Although browsers provide acoustic echo cancellation (AEC), loud speaker volume or physical room reverb can bleed through. The server's VAD engine may interpret this reflected sound as a user interruption, causing the agent to cut off its own speech prematurely. Inform users in your UI that using headphones provides the best conversational experience.

3. Concurrency Spikes and Burst Billing

If an application experiences sudden traffic surges, simultaneous voice sessions can quickly exhaust your account's concurrency limits. As detailed in the ElevenLabs Burst Pricing guide, the platform supports automatic burst scaling up to 3x your tier's baseline limit (capped at 300 concurrent calls for non-enterprise accounts). However, calls operating within this burst window are billed at double the standard rate (approximately $0.16 per minute). Configure concurrency thresholds in your dashboard to avoid unexpected billing spikes.

4. Client Tool Latency and Timeouts

When an agent invokes a client tool with expects_response: true, the user experiences a conversational pause while the browser executes the function and returns data. If your client tool performs slow network calls, heavy DOM calculations, or database writes, the assistant will pause awkwardly. Keep client tool functions lightweight and return simple JSON confirmation payloads within 200 to 400 milliseconds.

5. Mobile Background Throttling

Mobile operating systems aggressively suspend WebRTC connections and pause JavaScript timers when the user switches browser tabs or locks their device screen. Ensure your application listens to visibilitychange events (as detailed in the MDN Web Docs on the Page Visibility API) and cleanly invokes endSession() if the tab is hidden, preventing ghost sessions that consume active minutes.

Who Should Skip ElevenLabs Conversational AI: Architectural Fit and Alternatives

While integrated WebRTC voice platforms offer significant benefits for interactive web experiences, they are not appropriate for every project:

• Strict Air-Gapped and Zero-Cloud Environments: Highly regulated healthcare, defense, or banking systems that prohibit audio from traversing external public networks cannot use cloud-hosted voice APIs. These architectures require self-hosted, on-premises models running on private hardware.
• One-Way Batch Media Generation: If your application generates podcast episodes, audiobooks, automated video voiceovers, or static voicemail recordings, conversational agents introduce unnecessary architectural complexity. Use standard batch text-to-speech APIs instead.
• Asynchronous Audio Transcription: If your workload consists of transcribing recorded meeting files, customer call archives, or uploaded audio notes, you need an asynchronous speech-to-text batch pipeline, not a full-duplex WebRTC connection.
• Low-Margin, High-Volume Text Applications: Voice interactions carry distinct infrastructure costs (approximately $0.08 per minute). If your unit economics cannot support voice session expenses, stick to standard text-based chat completions.

Implementation Roadmap: What to Do Next

1. Sign up for an account and define a test agent in the dashboard with custom conversation flow settings (turn_timeout and turn_eagerness).
2. Implement the backend /api/voice-token route in your server application to keep your private API key secured.
3. Install @elevenlabs/react in your frontend project and implement granular hooks to isolate UI re-renders.
4. Define and test at least one client tool using useConversationClientTool to give your voice agent direct control over your application state.
5. Review concurrency limits in your platform settings to configure burst protection and call duration limits before releasing to production.

Worth a look: ElevenLabs suits developers and engineering teams building real-time interactive voice applications that require responsive turn-taking, multi-model LLM routing, and browser-driven tool calling, while teams requiring offline air-gapped voice processing or simple batch audio generation should look elsewhere.