ElevenAgents Flutter SDK

Official Flutter SDK for ElevenAgents. Build voice-enabled applications with real-time bidirectional audio communication powered by WebRTC via LiveKit.

Features

• Real-time Voice Communication - Full-duplex audio streaming with low latency
• Text Messaging - Send text messages and contextual updates during conversations
• Client Tools - Register device-side functions that agents can invoke
• Reactive State Management - Built on ChangeNotifier for Flutter-idiomatic patterns
• Comprehensive Callbacks - Fine-grained event handlers for all conversation events
• Feedback System - Built-in support for rating agent responses
• Type Safety - Complete Dart type definitions for all APIs
• Data Residency - Support for custom endpoints and regional deployments
• Production Ready - Built on LiveKit's proven WebRTC infrastructure

Examples

The example directory contains a full-featured demo application showing:

• Voice conversation with real-time audio
• Text messaging
• Mute/unmute controls
• Connection state management
• Feedback buttons
• Client tool implementation
• Conversation history display

Run the example:

cd example
flutter run

Installation

Add to your pubspec.yaml (see version badge above):

dependencies:
  elevenlabs_agents: ^[latest]

Install dependencies:

flutter pub get

Platform Configuration

iOS

Add microphone permission to ios/Runner/Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>This app needs microphone access for voice conversations</string>

Set minimum iOS version in ios/Podfile:

platform :ios, '13.0'

Android

Add permissions to android/app/src/main/AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

Set minimum SDK version in android/app/build.gradle:

android {
    defaultConfig {
        minSdkVersion 21
    }
}

Quick Start

Here's a minimal example showing basic voice conversation functionality:

import 'package:flutter/material.dart';
import 'package:elevenlabs_agents/elevenlabs_agents.dart';
import 'package:permission_handler/permission_handler.dart';

void main() {
  WidgetsFlutterBinding.ensureInitialized();
  runApp(const MaterialApp(home: VoiceAssistant()));
}

class VoiceAssistant extends StatefulWidget {
  const VoiceAssistant({super.key});

  @override
  State<VoiceAssistant> createState() => _VoiceAssistantState();
}

class _VoiceAssistantState extends State<VoiceAssistant> {
  final _messages = <String>[];
  late final _client = ConversationClient(
    callbacks: ConversationCallbacks(
      onConnect: ({required conversationId}) {
        print('Connected with ID: $conversationId');
      },
      onMessage: ({required message, required source}) {
        setState(() {
          _messages.add('${source.name}: $message');
        });
      },
      onModeChange: ({required mode}) {
        print('Mode changed: ${mode.name}');
      },
      onError: (message, [context]) {
        print('Error: $message');
      },
    ),
  )..addListener(() {
      setState(() {});
    });

  @override
  void initState() {
    super.initState();
    _requestMicrophonePermission();
  }

  Future<void> _requestMicrophonePermission() async {
    await Permission.microphone.request();
  }

  @override
  void dispose() {
    _client.dispose();
    super.dispose();
  }

  Future<void> _startConversation() async {
    try {
      await _client.startSession(
        agentId: 'your-agent-id-here',
        userId: 'user-123',
      );
    } catch (e) {
      print('Failed to start conversation: $e');
    }
  }

  Future<void> _endConversation() async {
    await _client.endSession();
  }

  @override
  Widget build(BuildContext context) {
    final (isConnected, isDisconnected) = (
      _client.status == ConversationStatus.connected,
      _client.status == ConversationStatus.disconnected,
    );

    return Scaffold(
      body: Column(
        children: [
          Container(
            padding: const EdgeInsets.all(16),
            color: isConnected ? Colors.green : Colors.grey,
            child: Row(
              mainAxisAlignment: MainAxisAlignment.center,
              children: [
                Text('Status: ${_client.status.name}'),
                if (isConnected && _client.isSpeaking) const Text(' - AI Agent Speaking'),
              ],
            ),
          ),
          Expanded(
            child: ListView.builder(
              itemCount: _messages.length,
              itemBuilder: (context, index) => ListTile(
                title: Text(_messages[index]),
              ),
            ),
          ),
          Padding(
            padding: const EdgeInsets.all(16),
            child: Column(
              children: [
                if (isDisconnected)
                  ElevatedButton(
                    onPressed: _startConversation,
                    child: const Text('Start Conversation'),
                  ),
                if (isConnected) ...[
                  ElevatedButton(
                    onPressed: _endConversation,
                    style: ElevatedButton.styleFrom(
                      backgroundColor: Colors.red,
                    ),
                    child: const Text('End Conversation'),
                  ),
                  const SizedBox(height: 16),
                  ElevatedButton.icon(
                    onPressed: () => _client.toggleMute(),
                    icon: Icon(_client.isMuted ? Icons.mic_off : Icons.mic),
                    label: Text(_client.isMuted ? 'Unmute' : 'Mute'),
                  ),
                ],
              ],
            ),
          ),
        ],
      ),
    );
  }
}

Core Concepts

Starting a Conversation

Public Agent

For publicly available agents, provide just the agent ID:

await client.startSession(
  agentId: 'your-public-agent-id',
  userId: 'user-123',
);

Private Agent

For private agents, generate a conversation token from your backend and pass it to the SDK:

// Get token from your backend
final token = await yourBackend.getConversationToken();

await client.startSession(
  conversationToken: token,
  userId: 'user-123',
);

Note: Generating a token requires an ElevenLabs API key. Never expose this key on the client, instead fetch it from a backend service.

Configuration Overrides

Customize agent behavior on a per-session basis:

await client.startSession(
  agentId: 'your-agent-id',
  overrides: ConversationOverrides(
    agent: AgentOverrides(
      firstMessage: 'Hello! How can I help you today?',
      prompt: 'You are a helpful customer service assistant...',
      language: 'en',
      temperature: 0.7,
      maxTokens: 1000,
    ),
    tts: TtsOverrides(
      voiceId: 'custom-voice-id',
      stability: 0.5,
      similarityBoost: 0.8,
      style: 0.0,
      useSpeakerBoost: true,
    ),
    conversation: ConversationSettingsOverrides(
      maxDurationSeconds: 600,
      turnTimeoutSeconds: 10,
      textOnly: false,
    ),
  ),
  dynamicVariables: {
    'user_name': 'Alice',
    'account_tier': 'premium',
  },
);

Sending Messages

Send text messages and contextual updates during a conversation:

// Send a text message to the agent
client.sendUserMessage('I need help with my order');

// Send contextual information (invisible to user, visible to agent)
client.sendContextualUpdate('User is on order #12345 page');

// Send user activity signal, which will prevent the agent from speaking for ~2 seconds
// Useful for when a user is e.g. typing a message
client.sendUserActivity();

Microphone Control

// Mute the microphone
await client.setMicMuted(true);

// Unmute the microphone
await client.setMicMuted(false);

// Toggle mute state
await client.toggleMute();

// Check current mute state
if (client.isMuted) {
  print('Microphone is muted');
}

Feedback System

Allow users to rate agent responses:

// In your UI, show feedback buttons when available
if (client.canSendFeedback) {
  // User taps thumbs up
  client.sendFeedback(isPositive: true);

  // Or thumbs down
  client.sendFeedback(isPositive: false);
}

// Listen for feedback state changes
ConversationClient(
  callbacks: ConversationCallbacks(
    onCanSendFeedbackChange: ({required canSendFeedback}) {
      setState(() {
        // Update UI to show/hide feedback buttons
      });
    },
  ),
);

Client Tools

Register client-side tools that the agent can invoke to access device capabilities:

// Define a tool
class GetLocationTool implements ClientTool {
  @override
  Future<ClientToolResult?> execute(Map<String, dynamic> parameters) async {
    try {
      final location = await Geolocator.getCurrentPosition();

      return ClientToolResult.success({
        'latitude': location.latitude,
        'longitude': location.longitude,
        'accuracy': location.accuracy,
      });
    } catch (e) {
      return ClientToolResult.failure('Failed to get location: $e');
    }
  }
}

class LogMessageTool implements ClientTool {
  @override
  Future<ClientToolResult?> execute(Map<String, dynamic> parameters) async {
    final message = parameters['message'] as String?;

    if (message == null || message.isEmpty) {
      return ClientToolResult.failure('Missing message parameter');
    }

    print('Agent logged: $message');

    // For tools where no response is needed, don't return anything
  }
}

// Register tools with the client
final client = ConversationClient(
  clientTools: {
    'getUserLocation': GetLocationTool(),
    'logMessage': LogMessageTool(),
  },
  callbacks: ConversationCallbacks(
    onUnhandledClientToolCall: (toolCall) {
      print('Agent called unimplemented tool: ${toolCall.toolName}');
      print('Parameters: ${toolCall.parameters}');
    },
  ),
);

Tool execution flow:

1. Agent decides to invoke a tool
2. SDK receives tool call request
3. SDK looks up and executes the registered tool
4. Tool returns success or failure result
5. SDK sends result back to agent
6. Agent continues conversation with the result

Callbacks

The SDK provides comprehensive callbacks for all conversation events:

ConversationClient(
  callbacks: ConversationCallbacks(
    // Connection lifecycle
    onConnect: ({required conversationId}) {
      print('Connected: $conversationId');
    },
    onDisconnect: (details) {
      print('Disconnected: ${details.reason}');
    },
    onStatusChange: ({required status}) {
      print('Status: ${status.name}');
    },
    onError: (message, [context]) {
      print('Error: $message');
    },

    // Messages and transcripts
    onMessage: ({required message, required source}) {
      print('[${source.name}] $message');
    },
    onModeChange: ({required mode}) {
      // Called when conversation mode changes
      if (mode == ConversationMode.speaking) {
        print('Agent started speaking');
      } else {
        print('Agent is listening');
      }
    },
    onTentativeUserTranscript: ({required transcript, required eventId}) {
      // Real-time transcription as user speaks
      print('User speaking: $transcript');
    },
    onUserTranscript: ({required transcript, required eventId}) {
      // Finalized user transcription
      print('User said: $transcript');
    },
    onTentativeAgentResponse: ({required response}) {
      // Agent's streaming text response
      print('Agent composing: $response');
    },
    onAgentResponseCorrection: (correction) {
      // When agent corrects its response
      print('Agent corrected: $correction');
    },

    // Conversation state
    onConversationMetadata: (metadata) {
      print('Conversation ID: ${metadata.conversationId}');
      print('Audio formats: ${metadata.agentOutputAudioFormat}');
    },

    // Audio and voice activity
    onVadScore: ({required vadScore}) {
      // Voice activity detection score (0.0 to 1.0)
      print('VAD score: $vadScore');
    },
    onInterruption: (event) {
      print('User interrupted agent');
    },

    // Feedback
    onCanSendFeedbackChange: ({required canSendFeedback}) {
      // Show/hide feedback buttons
      setState(() {});
    },

    // Tools
    onUnhandledClientToolCall: (toolCall) {
      print('Unhandled tool: ${toolCall.toolName}');
    },
    onAgentToolResponse: (response) {
      print('Tool ${response.toolName} executed');
    },

    // MCP (Model Context Protocol)
    onMcpToolCall: (toolCall) {
      print('MCP tool: ${toolCall.toolName}');
    },
    onMcpConnectionStatus: (status) {
      print('MCP integrations: ${status.integrations.length}');
    },

    // ASR (Automatic Speech Recognition)
    onAsrInitiationMetadata: (metadata) {
      print('ASR metadata: ${metadata}');
    },

    // Streaming response parts
    onAgentChatResponsePart: (part) {
      // Streaming text chunks: start, delta, or stop
      print('Agent text [${part.type}]: ${part.text}');
    },

    // Debug (all raw events), very noisy
    onDebug: (data) {
      print('Debug: $data');
    },
  ),
);

Reactive State Management

The ConversationClient extends ChangeNotifier, making it easy to integrate with Flutter's reactive patterns:

class _MyWidgetState extends State<MyWidget> {
  late ConversationClient _client;

  @override
  void initState() {
    super.initState();
    _client = ConversationClient();

    // Listen to all state changes
    _client.addListener(_onClientStateChanged);
  }

  void _onClientStateChanged() {
    setState(() {
      // Widget rebuilds when client state changes
    });
  }

  @override
  void dispose() {
    _client.removeListener(_onClientStateChanged);
    _client.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    // Access reactive state properties
    return Column(
      children: [
        Text('Status: ${_client.status.name}'),
        Text('Speaking: ${_client.isSpeaking}'),
        Text('Muted: ${_client.isMuted}'),
        if (_client.conversationId != null)
          Text('ID: ${_client.conversationId}'),
        if (_client.canSendFeedback)
          Row(
            children: [
              IconButton(
                icon: const Icon(Icons.thumb_up),
                onPressed: () => _client.sendFeedback(isPositive: true),
              ),
              IconButton(
                icon: const Icon(Icons.thumb_down),
                onPressed: () => _client.sendFeedback(isPositive: false),
              ),
            ],
          ),
      ],
    );
  }
}

Regional Deployments and Data Residency

For self-hosted deployments or region-specific requirements:

final client = ConversationClient(
  apiEndpoint: 'https://api.eu.elevenlabs.io',
  websocketUrl: 'wss://livekit.rtc.eu.elevenlabs.io',
);

Important: Both the API endpoint and WebSocket URL must point to the same geographic region to avoid authentication errors.

API Reference

ConversationClient

Constructor

ConversationClient({
  String? apiEndpoint,  // Default: 'https://api.elevenlabs.io'
  String? websocketUrl, // Default: 'wss://livekit.rtc.elevenlabs.io'
  ConversationCallbacks? callbacks,
  Map<String, ClientTool>? clientTools,
})

Properties

Methods

startSession Parameters

* Either agentId or conversationToken must be provided

Enums

ConversationStatus

• disconnected - Not connected to any agent
• connecting - Connection in progress
• connected - Active conversation
• disconnecting - Disconnect in progress

ConversationMode

• listening - Agent is listening to user
• speaking - Agent is speaking

Role

• user - Message from the user
• ai - Message from the agent

Troubleshooting

Microphone Permission Denied

Ensure permissions are properly configured in platform files and granted by the user.

iOS: Check Info.plist has NSMicrophoneUsageDescription Android: Check AndroidManifest.xml has RECORD_AUDIO permission

Connection Failures

• Verify your agent ID or conversation token is correct
• Check network connectivity and firewall settings
• Ensure WebRTC ports are not blocked (UDP 3478-3479, TCP 443)
• For private agents, verify your backend token generation is correct

Poor Audio Quality

• Check microphone permissions
• Verify device microphone is working in other apps
• Check network bandwidth (voice requires steady connection)
• Try on a different network to rule out firewall issues

Agent Not Responding

• Verify the agent is properly configured in your ElevenLabs dashboard
• Check that the agent has appropriate tools and knowledge base
• Monitor the onDebug callback for detailed event logs
• Check onError callback for specific error messages

Tool Calls Not Working

• Ensure tool names match exactly between agent config and client registration
• Verify ClientTool implementations return proper ClientToolResult
• Check onUnhandledClientToolCall for tools the agent tried to call
• Use onDebug to see the raw tool call messages

Testing

# Run tests
flutter test

# Run integration tests
flutter test integration_test/

# Run with coverage
flutter test --coverage

Platform Support

Releasing

To release a new version of the SDK, use the Claude Code release command:

claude /release <version>

This will update version strings, CHANGELOG, run checks, open a PR, and (after merge) tag the release to trigger publishing to pub.dev.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Support

• Documentation: https://elevenlabs.io/docs
• Discord Community: https://discord.gg/elevenlabs
• Email Support: support@elevenlabs.io
• Issue Tracker: GitHub Issues

Related Projects

• ElevenLabs Android SDK
• ElevenLabs React Native SDK
• LiveKit Flutter SDK
• ElevenLabs API Documentation