"AI Voice Agents for FreeSWITCH: ESL Integration Explained"

FreeSWITCH is the platform people reach for when they need scale and control: high concurrency, clean media handling, and an event system that lets you script call behavior precisely. Those same strengths make it an excellent host for an AI voice agent. You do not replace FreeSWITCH with an AI platform; you stream call audio to the agent and let it drive the call through the Event Socket Layer.

This guide explains the integration in FreeSWITCH terms: media over mod_audio_stream, control over ESL, and how to keep concurrency healthy as you grow.

The shape of the integration

At a high level there are two channels of communication between FreeSWITCH and the AI:

1. Media — the live call audio, streamed to the agent and back.
2. Control — the instructions to transfer, bridge or hang up, driven by ESL.

Keeping media and control separate is idiomatic FreeSWITCH, and it maps neatly onto how an AI agent works. The agent listens to the media stream, decides what to do, and expresses those decisions as ESL events and commands.

Media: mod_audio_stream over WebSocket

The cleanest way to get call audio out of FreeSWITCH is mod_audio_stream, which opens a WebSocket and pushes call audio to a remote endpoint in real time. In the dialplan, you start the stream and include the agent's dial code in the URL so the platform knows which configuration to use:

<extension name="ai_agent">
  <condition field="destination_number" expression="^85(\d{3})$">
    <action application="answer"/>
    <action application="audio_stream" data="start wss://app.bridgeside.ai/fs/$1 mono 8k"/>
    <action application="park"/>
  </condition>
</extension>

The audio_stream action begins streaming; park holds the call open while the conversation happens. The platform receives the audio, runs its speech-to-text, language model and text-to-speech pipeline, and streams synthesized speech back over the same socket. Because the media is handled as a stream rather than a re-dial, turn-taking stays responsive.

If you prefer RTP to WebSocket, FreeSWITCH can also bridge media directly, but mod_audio_stream is usually the least-friction path and plays well with cloud endpoints.

Control: the Event Socket Layer

ESL is where FreeSWITCH shines for this use case. The platform holds an ESL connection (inbound or outbound mode) and uses it to observe call state and issue commands.

• Transfer to a human. When the agent qualifies a lead, it issues an ESL command to transfer or bridge the parked call to a human destination — an agent extension, a gateway, or a queue. Because the call is parked and live, the human joins a conversation that is already connected.
• Hangup. When the conversation ends, the agent issues a uuid_kill (or a graceful hangup with a cause code) over ESL, and your normal CDR and post-call logic run.
• State awareness. ESL events tell the platform when the channel answers, when media is flowing, and when the far end hangs up, so the agent never talks into a dead line.

Outbound ESL mode is particularly clean for this: FreeSWITCH connects out to a socket server per call, which the platform accepts, giving each call its own control channel with no shared state to manage.

Concurrency at scale

FreeSWITCH is chosen for concurrency, so it is worth being explicit about how AI calls scale.

Each AI call consumes one media stream and one control channel. There is no shared serialization point in the integration, so concurrent capacity is bounded by three things:

1. CPU and bandwidth on the FreeSWITCH box, mostly for media handling.
2. Your network egress to the platform for the audio streams.
3. Your platform plan's concurrent-call ceiling.

To keep things healthy at high concurrency:

• Pin mod_audio_stream to 8 kHz mono unless you have a specific reason for wideband; it halves bandwidth and matches telephone-quality audio.
• Watch RTP and socket file-descriptor limits on the host and raise ulimit accordingly.
• Spread very large deployments across multiple FreeSWITCH instances behind your existing routing, exactly as you would for human traffic.

Because the agent is per-call, adding capacity is horizontal: more FreeSWITCH, more streams, no central bottleneck to re-tune.

Recording and transcripts

FreeSWITCH's record_session captures the full call, AI side included, since the AI is simply the media peer. You keep recordings where you already keep them, in the format you already use. The platform additionally stores a per-call transcript, which is invaluable for QA, compliance review and finding patterns across large call volumes.

Dispositions and downstream systems

For outbound work, each call should produce a structured outcome. The platform posts the result — outcome, captured fields, and any do-not-call request — to a webhook or your CRM. Treat the DNC signal as first-class: if a lead asks to be removed, that must flow into your suppression list immediately, before the next campaign runs.

Security notes

• Terminate the media WebSocket over wss:// so audio is encrypted in transit.
• Restrict ESL to the platform's addresses and use a strong ESL password; ESL is powerful and should never be open.
• Scope the platform's ESL permissions to the commands it needs: transfer, bridge, hangup, and event subscription.

When FreeSWITCH is the right host

If you are already on FreeSWITCH, keep it. Its media flexibility and ESL control make it arguably the nicest place to run an AI voice agent, and the integration is entirely additive. If you are choosing between platforms specifically to run AI, FreeSWITCH is a strong pick when you expect high concurrency or want fine-grained, event-driven control of call behavior.

Rollout checklist

• Load and configure mod_audio_stream.
• Add the dialplan extension that starts the stream and parks the call.
• Stand up the ESL connection (outbound mode recommended) with a scoped, strong password.
• Confirm a test call streams audio both ways and transfers cleanly to a human.
• Verify recording captures the full call and transcripts appear.
• Wire dispositions and DNC suppression.
• Load-test at your expected peak concurrency and tune host limits.

Add the media stream, add the ESL control, and FreeSWITCH does what it already does best while the agent handles the conversation. Nothing about your routing, trunking or scale strategy has to change.

If you run FreeSWITCH and want to trial an agent, contact us and we will help you wire up mod_audio_stream and ESL on a test extension.