Table of contents · 8 sections

Build a Real-Time Voice Agent in React with ElevenLabs

Traditional text chat interfaces force users into tedious typing loops, while custom voice stacks built by daisy-chaining separate speech-to-text, LLM endpoints, and text-to-speech services suffer from jarring multi-second latency and fragile state synchronization. ElevenLabs Conversational AI unifies real-time speech processing, conversational turn-taking, and client-side tool execution through a streamlined WebSocket connection (with WebRTC supported via session tokens). In this guide, you will build and deploy a production-ready voice assistant in React that executes client-side actions and mutates UI components with sub-second response times.

Some links in this post are affiliate links; see our disclosure.

---

Quick Verdict: Pick It or Skip It?

• Pick it if:
  • You need sub-second, natural voice interaction directly inside a web browser or mobile client.
  • You want your voice agent to trigger client-side actions—such as navigating routes, opening modals, or updating form inputs—without round-tripping through external backend webhooks.
  • You prefer an all-in-one conversational runtime that handles voice activity detection (VAD), turn-taking, and barge-in interruptions out of the box.
• Skip it if:
  • Your product operates in offline, air-gapped, or strictly on-premise environments where third-party cloud streaming is prohibited.
  • Your user workflows are purely asynchronous or text-heavy (like code reviews or complex document authoring) where spoken dialogue adds friction.
  • Your infrastructure budget cannot absorb metered per-minute voice audio billing alongside standard LLM token consumption.

---

What ElevenLabs Conversational AI React Is (and What It Replaces)

Historically, building a voice-enabled interface required connecting three separate services:

1. Speech-to-Text (STT): An automatic speech recognition WebSocket or batch transcription endpoint that converts microphone audio into text tokens.
2. Language Model (LLM): A reasoning engine (such as GPT-4o, Claude 3.5 Sonnet, or a self-hosted open-weights model) that receives text prompts and generates responses.
3. Text-to-Speech (TTS): A speech synthesizer that parses the LLM token stream and renders audio frames back to the client.

This daisy-chain architecture introduces compounding latency. Each stage requires independent network serialization, buffer delays while waiting for complete grammatical clauses, and custom interruption logic to halt audio playback whenever a user speaks up mid-sentence. When an interruption occurs, developers must manually cancel inflight LLM generation, purge audio playback buffers, and reset state.

ElevenLabs Conversational AI replaces this pipeline with a single, bidirectional conversational runtime. Speech recognition, turn-taking orchestration, interruption handling, and neural audio generation run inside a unified cloud session. The client application establishes an authenticated session over WebRTC or WebSockets, streaming raw microphone buffers and receiving low-latency audio streams directly.

Crucially, the platform includes a native Client Tools subsystem. While traditional agent tooling relies on server-side webhooks to call remote databases or external APIs, client tools execute directly inside the user's browser runtime. The AI agent can decide to invoke a tool, pass validated arguments to your frontend React components, wait for your JavaScript handler to execute, and incorporate the returned data into its ongoing vocal dialogue.

Architectural Dimension	Stitched Multi-Vendor Pipeline (STT + LLM + TTS)	ElevenLabs Conversational AI Platform
Typical Round-Trip Latency	1,800 ms – 3,500 ms (compounded across three API hops)	500 ms – 900 ms (unified edge streaming via WebSocket or WebRTC)
Barge-in / Interruption	Manual (requires custom VAD, socket cancellation, and audio buffer flushes)	Automatic (native server-side turn-taking and speech interruption)
Client-Side Automation	High complexity (requires custom socket events to bridge LLM tool calls to the DOM)	Built-in (native clientTools schema registered directly in the React SDK)
Server Infrastructure	Heavy (orchestration servers, WebSocket proxies, state managers)	Minimal (lightweight server endpoint to issue ephemeral signed session tokens)
Transport Protocol	Fragmented (mixed REST calls and multiple concurrent WebSockets)	Unified (low-latency WebSocket via signed URL or WebRTC via session token)
Billing Model	Multiple invoices (separate STT per minute, LLM tokens, and TTS characters)	Unified metered minutes (~$0.08 to $0.10/min) plus underlying LLM costs

---

Why Turn-Based Voice Matters Now

Voice interfaces have moved beyond rigid command-and-control bots. Users expect natural, full-duplex conversations where human and artificial agents exchange communication cues continuously. If a user interjects with "Wait, go back," the synthetic voice must cut off instantly without droning on.

Building this level of responsiveness on the web is notoriously difficult due to three core browser audio constraints:

• Acoustic Echo Cancellation (AEC): The browser microphone easily picks up the agent's synthetic voice emitting from laptop speakers, creating self-feedback loops and false interruptions.
• Contextual UI Synchronization: Voice should not exist in an isolated bubble. When a user asks to view invoices or change display modes, the DOM must update in tandem with spoken dialogue.
• Security at the Edge: Production API credentials cannot be bundled into client-side JavaScript. Voice sessions require on-demand authentication using short-lived tokens.

The official @elevenlabs/react SDK resolves these challenges by wrapping connection state, audio device pipelines, and client-side tool handlers into idiomatic React hooks. Rather than managing raw audio contexts and WebRTC peer connections by hand, frontend teams treat voice interactions as reactive state.

---

How ElevenLabs Conversational AI React Works: Architecture, Transport, and Client Tools

Architecting an interactive voice app requires dividing responsibilities cleanly between your backend server, the client browser, and the ElevenLabs Conversational AI cloud infrastructure.

The runtime flow consists of three distinct phases:

1. Server-Side Authentication Handshake: The client browser never receives your master ElevenLabs API key. When a user requests a conversation, your React app makes an internal HTTP request to your backend. The backend calls the ElevenLabs Get Signed URL API using your secret xi-api-key. This endpoint returns an ephemeral WebSocket URL (wss://) containing an embedded single-use token (whereas WebRTC transport requires session tokens issued via GET /v1/convai/conversation/token).
2. Media Stream Initialization: The frontend receives the ephemeral signed_url and passes it to startSession({ signedUrl }). Before connecting, the browser requests microphone access via navigator.mediaDevices.getUserMedia({ audio: true }). Once granted, the SDK initiates a full-duplex WebSocket connection (wss://) with the ElevenLabs edge gateway (alternatively, passing a session token via conversationToken establishes a WebRTC connection).
3. Turn-Taking and Dynamic Tool Invocations: As the user speaks, audio frames are analyzed in real time. When the conversational model determines that an intent requires a frontend action (such as updateInterfaceTheme), it sends an execution frame to the client. The SDK catches this event, executes the local JavaScript function registered under clientTools, and—if the tool is configured with expects_response: true—delivers the return value back to the agent to synthesize into the next spoken utterance.

---

Pros and Cons of ElevenLabs Conversational AI React

Evaluating ElevenLabs Conversational AI for frontend web applications requires balancing its ultra-low latency and developer ergonomics against cloud audio costs and browser runtime constraints:

Pros

• Sub-Second Conversational Latency: Bypasses multi-hop STT-to-LLM-to-TTS pipeline latency by orchestrating speech processing in a single unified session, achieving 500 ms – 900 ms turnaround.
• Native Client-Side Tool Calling: The clientTools interface allows the voice agent to trigger local JavaScript functions, modify React state, and navigate routes directly in the browser without backend hops.
• Automatic Turn-Taking and Interruption (Barge-in): Built-in voice activity detection (VAD) instantly pauses synthetic speech when the user speaks, delivering human-like conversational flow out of the box.
• Idiomatic React Integration: The official @elevenlabs/react package exposes clean hooks (useConversation, useConversationControls, useConversationStatus) that seamlessly bind voice state to reactive UI components.
• High-Fidelity Neural Voices: Leverages ElevenLabs' industry-leading voice library and customizable agent personas with consistent pronunciation across dozens of languages.

Cons

• Metered Per-Minute Pricing: Voice sessions incur runtime charges (~$0.08 to $0.10/min) in addition to underlying LLM token costs, making unmonetized or casual traffic expensive at scale.
• Cloud & Network Dependency: Requires an active, stable internet connection; cannot run locally in air-gapped, zero-cloud, or strictly on-premise environments.
• Audio Permission & Autoplay Constraints: Strict browser security policies require secure contexts (https or localhost) and an explicit user gesture to initialize microphone streams and audio playback.
• Vulnerability to Slow Client Tools: If a client tool handler takes too long to resolve when expects_response: true is configured, the voice agent pauses speech, requiring developers to enforce strict client-side timeouts.

---

Step-by-Step Walkthrough: Building an ElevenLabs Conversational AI React Voice Agent

Let us build an authenticated, interactive voice agent inside a modern React application using Next.js (App Router) for token generation and @elevenlabs/react on the frontend.

Step 1: Install Dependencies

In your project directory, install the official ElevenLabs React package:

npm install @elevenlabs/react

As documented in the official repository, @elevenlabs/react automatically re-exports all core utilities from @elevenlabs/client, eliminating the need to install the client package separately.

Step 2: Generating Ephemeral Signed URLs on the Server

To keep credentials secure, create an API route that queries the ElevenLabs authentication gateway. In Next.js, add the following route handler at app/api/voice-token/route.ts:

// app/api/voice-token/route.ts
import { NextResponse } from "next/server";

export async function GET() {
  const agentId = process.env.ELEVENLABS_AGENT_ID;
  const apiKey = process.env.ELEVENLABS_API_KEY;

  if (!agentId || !apiKey) {
    return NextResponse.json({ error: "Missing ElevenLabs credentials" }, { status: 500 });
  }

  try {
    const url = new URL("https://api.elevenlabs.io/v1/convai/conversation/get-signed-url");
    url.searchParams.set("agent_id", agentId);

    const response = await fetch(url.toString(), {
      method: "GET",
      headers: { "xi-api-key": apiKey },
      cache: "no-store",
    });

    if (!response.ok) {
      return NextResponse.json({ error: "Failed to generate signed URL" }, { status: response.status });
    }

    const data: { signed_url: string } = await response.json();
    return NextResponse.json({ signedUrl: data.signed_url });
  } catch (err: unknown) {
    const message = err instanceof Error ? err.message : "Internal Server Error";
    return NextResponse.json({ error: message }, { status: 500 });
  }
}

This route ensures your private xi-api-key remains on the server while supplying single-use signed URLs to client components on demand.

Step 3: Configuring the Client Tool Schema in ElevenLabs

Before your agent can invoke a client-side function, configure its schema via the ElevenLabs dashboard under Agents > Tools > Add Tool > Client, or programmatically using the ElevenLabs CLI.

Here is the exact JSON configuration schema for a client tool named updateInterfaceTheme:

{
  "type": "client",
  "name": "updateInterfaceTheme",
  "description": "Switches the user interface theme between dark and light modes according to user preference.",
  "expects_response": true,
  "parameters": {
    "type": "object",
    "properties": {
      "theme": {
        "type": "string",
        "enum": ["dark", "light"],
        "description": "The desired theme style requested by the user."
      }
    },
    "required": ["theme"]
  }
}

When expects_response is set to true, the voice engine pauses speech output until your React handler completes execution and returns a message, which is then fed into the model's context.

Step 4: Wrapping the App in ConversationProvider

As detailed in the ElevenLabs SDK Architecture Guide, all conversation hooks strictly require an ancestor <ConversationProvider> to manage conversation connection state.

Wrap your component hierarchy in app/voice/layout.tsx:

// app/voice/layout.tsx
"use client";

import { ConversationProvider } from "@elevenlabs/react";
import React from "react";

export default function VoiceLayout({ children }: { children: React.ReactNode }) {
  return (
    <ConversationProvider>
      <div className="voice-wrapper">{children}</div>
    </ConversationProvider>
  );
}

Step 5: Building the Voice Assistant with Client Tools

Now, create the interactive client component at app/voice/page.tsx. This component uses useConversation() to manage session state and registers the client-side tool:

// app/voice/page.tsx
"use client";

import React, { useState, useCallback } from "react";
import { useConversation } from "@elevenlabs/react";

export default function VoiceAgentPage() {
  const [activeTheme, setActiveTheme] = useState<"dark" | "light">("dark");
  const [errorMessage, setErrorMessage] = useState<string | null>(null);

  const conversation = useConversation({
    onConnect: () => setErrorMessage(null),
    onError: (err) => setErrorMessage(typeof err === "string" ? err : "Voice session error"),
  });

  const handleStartConversation = useCallback(async () => {
    try {
      setErrorMessage(null);

      // Verify and request microphone permissions via MDN MediaDevices API
      if (typeof window !== "undefined" && navigator.mediaDevices) {
        const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
        // Stop permission-checking tracks to prevent a dangling microphone stream before SDK initialization
        stream.getTracks().forEach((track) => track.stop());
      } else {
        throw new Error("Microphone API unavailable in this browser environment.");
      }

      // Fetch the ephemeral signed URL from our internal Next.js backend
      const res = await fetch("/api/voice-token");
      if (!res.ok) throw new Error("Failed to acquire voice session token.");
      const { signedUrl } = await res.json();

      // Start session with client tool handler
      conversation.startSession({
        signedUrl,
        clientTools: {
          updateInterfaceTheme: async ({ theme }: { theme: "dark" | "light" }) => {
            setActiveTheme(theme);
            return `Switched interface theme to ${theme} mode.`;
          },
        },
      });
    } catch (err: unknown) {
      const msg = err instanceof Error ? err.message : "Failed to connect to agent";
      setErrorMessage(msg);
    }
  }, [conversation]);

  const isConnected = conversation.status === "connected";
  const isConnecting = conversation.status === "connecting";

  return (
    <div style={{ padding: "2rem", background: activeTheme === "dark" ? "#0f172a" : "#ffffff", color: activeTheme === "dark" ? "#f8fafc" : "#0f172a" }}>
      <h2>Voice Assistant ({conversation.status})</h2>
      <p>Say &quot;Switch to light mode&quot; to trigger client-side React state mutations directly from voice.</p>

      {isConnected && <p>Agent is {conversation.isSpeaking ? "speaking..." : "listening..."}</p>}
      {errorMessage && <p style={{ color: "red" }}>{errorMessage}</p>}

      {!isConnected ? (
        <button onClick={handleStartConversation} disabled={isConnecting}>
          {isConnecting ? "Connecting..." : "Start Conversation"}
        </button>
      ) : (
        <button onClick={() => conversation.endSession()}>End Conversation</button>
      )}
    </div>
  );
}

If you are packaging your voice assistant into a commercial product, pair it with the AI SaaS landing template to quickly deploy a high-converting web frontend.

---

Pitfalls, Operational Limits, and Pricing Reality

Deploying voice agents to production web environments involves engineering trade-offs that rarely show up in local tests:

1. Browser Microphone Permissions & Secure Contexts

As detailed on MDN Web Docs regarding MediaDevices, the getUserMedia API is strictly locked to secure contexts (https:// or localhost). Testing over an unencrypted IP causes navigator.mediaDevices to evaluate to undefined, halting initialization. Furthermore, browsers enforce strict autoplay policies: audio playback cannot begin without an initial explicit user gesture like clicking a button.

2. Audio Processing Re-renders & Component Isolation

In earlier SDK versions, developers bound useConversation at the top level of large component trees. Because useConversation updates state across microphone levels and connection states, frequent changes trigger excessive re-renders.

In production, leverage granular hooks:

• useConversationControls(): Contains only the static startSession and endSession callbacks; never triggers re-renders on audio flux.
• useConversationStatus(): Only re-renders when connection status transitions between 'disconnected', 'connecting', and 'connected'.
• Isolate visual audio meters (getInputByteFrequencyData) into small leaf-node canvas components to keep the main application tree unburdened.

3. Tool Execution Latency & Timeouts

When an agent invokes a client tool with expects_response: true, it pauses vocal delivery while awaiting your return promise. If your client tool triggers a slow asynchronous query, the voice bot sits in uncomfortable silence. Always enforce a strict client-side timeout (e.g., 2,500 ms) on asynchronous client tool functions.

4. Pricing Structure and Billing Realities (Pricing as of October 2026)

Voice agents carry higher ongoing costs than static text chat interfaces. ElevenLabs prices its service using base subscription tiers combined with metered conversational minutes (Pricing as of October 2026):

• Free Plan: $0/month (10,000 monthly character credits, ~10 minutes, non-commercial).
• Starter Plan: $6/month ($5/mo billed annually, 30,000 credits).
• Creator Plan: $22/month ($18.33/mo billed annually, 121,000 credits).
• Pro Plan: $99/month ($82.50/mo billed annually, 600,000 credits).
• Scale Plan: $299/month ($249.17/mo billed annually, 1.8M (1,800,000) credits, 30 concurrent calls).
• Business Plan: $990/month ($825/mo billed annually, 6M (6,000,000) credits, 40 concurrent calls).

Active Conversational AI sessions are metered at approximately $0.08 to $0.10 per minute, depending on plan tier. ElevenLabs provides an automatic silence discount of approximately 95% on audio charges when users remain quiet for intervals exceeding 10 seconds. External LLM token costs (e.g., custom Claude or OpenAI endpoints) are billed separately.

How Credit Allowances Translate to Conversational Call Minutes

In ElevenLabs Conversational AI, credits translate directly into active voice conversation time at an approximate baseline rate of 1,000 credits per active conversational minute:

• Starter (30,000 credits): ~30 conversation minutes per month.
• Creator (121,000 credits): ~121 conversation minutes (~2 hours) per month.
• Pro (600,000 credits): ~600 conversation minutes (~10 hours) per month.
• Scale (1.8M credits): ~1,800 conversation minutes (~30 hours) per month.
• Business (6M credits): ~6,000 conversation minutes (~100 hours) per month.

Because the system applies an automatic ~95% silence discount for pauses longer than 10 seconds, sessions with intermittent user thinking or listening periods often consume significantly fewer credits per clock minute than continuous audio streaming.

Workload-to-Plan Matching Recommendations

Selecting the right tier depends on whether your workload is exploratory prototyping or a scaled production SaaS:

• Prototyping & Local Development (Free or Starter): The Free (10,000 credits) and Starter (30,000 credits) tiers are ideal for experimenting with the @elevenlabs/react SDK, testing client-side tool schemas, and local UI integration without upfront commitments.
• Early-Stage MVPs & Private Betas (Creator): The Creator plan (121,000 credits) provides roughly 2 hours of live conversational audio monthly. This is well-suited for internal demos, founder-led user interviews, or validating voice workflows with an initial cohort of test users.
• Production SaaS Applications (Pro to Scale): Commercial SaaS applications with active daily users require the Pro plan (600,000 credits, ~10 hours of talk time) at a minimum. When user traffic scales and multiple users converse simultaneously, transitioning to the Scale plan (1.8M credits, 30 concurrent calls) becomes necessary to avoid concurrency limits and ensure high availability.
• High-Throughput Enterprise Deployments (Business): Enterprise customer support bots, inbound call center automation, and mission-critical voice agents should deploy on the Business plan (6M credits, 40 concurrent calls) to leverage higher concurrency caps, enterprise-grade volume, and volume-discounted overages.

---

Who Should Build This (and Who Should Skip It)

Conversational voice agents are powerful, but they are not the correct interface pattern for every product.

Who Should Build This

• Interactive SaaS Workflows: Products where hands-free navigation speeds up tasks, such as medical charting, warehouse logistics, or field data collection.
• Language Learning & Coaching Apps: Applications requiring immediate phoneme correction, active pronunciation feedback, and natural verbal dialogue.
• High-Touch Customer Concierges: E-commerce and finance apps where client tools filter inventory, highlight items, or guide checkouts in real time.

Who Should Skip It

• Dense Technical / Code Generation Tools: Developers working with complex codebases or nested JSON structures need visual diffs and terminal copy-paste buffers, not a spoken voice reading syntax aloud.
• High-Concurrency, Zero-Budget Hobby Projects: If your application cannot monetize users or justify ongoing costs, free-tier limits will be exhausted quickly.
• Strictly Regulated, Offline Environments: If your app must function in submarines or air-gapped facilities with no internet, rely on embedded local models (whisper.cpp and Piper TTS) on local silicon instead of cloud APIs.

Worth a look: ElevenLabs fits teams building production web or mobile applications that need sub-second conversational voice and seamless client-side UI control, but developers building purely text-based agents or teams requiring fully air-gapped, offline voice pipelines should skip it.

---

What to Do Next

1. Create a free developer account on the ElevenLabs Console and create a basic conversational agent persona.
2. Define your first client-side tool under Agent Settings > Tools, ensuring your parameter names and data types match your frontend TypeScript interfaces.
3. Add the Next.js /api/voice-token route to your backend repository, setting ELEVENLABS_API_KEY and ELEVENLABS_AGENT_ID in your .env.local configuration.
4. Mount the @elevenlabs/react <ConversationProvider> at your application layout level and test microphone permissions on a secure https:// or localhost connection.
5. Implement client-side error boundaries and fallback UI states for users who deny microphone access or navigate using incompatible web browsers.