Bridgeside AI
← All posts

FreeSWITCH AI Agent Architecture for Call Centers

How to design a FreeSWITCH AI agent for a call center, from media streaming versus SIP to a bot, to ESL security, dialplan placement, FusionPBX, scaling and monitoring.

A FreeSWITCH AI agent in a call center is more than a WebSocket and a language model. It has to sit in the right place in the dialplan, hand callers to people without dropping them, survive a switch restart, and scale with the rest of your traffic. This guide is about those architecture decisions: how audio should reach the agent, how the agent controls the call, where the extension goes, what changes on FusionPBX, and what to watch once it is live.

If you want the module-level details, read the FreeSWITCH mod_audio_stream guide. For a shorter walkthrough of media and control, see running an AI phone agent on FreeSWITCH with ESL.

Two ways to build a FreeSWITCH AI agent

There are two broad ways to put an AI on a FreeSWITCH call.

SIP to a bot

The switch bridges the caller to a SIP endpoint that runs the AI. The bot is just another destination: you bridge to it like a gateway or a phone.

This is simple to describe and works with any SIP-capable AI service, but it has costs:

  • The call leaves your control. Once bridged, the bot owns its leg. To transfer, the bot must send a SIP REFER or ask you to re-route, and your switch must handle that well.
  • Two media legs. Codec negotiation, NAT and RTP between your switch and the bot become another path to debug.
  • Context travels badly. Lead IDs and campaign data have to ride in SIP headers that both sides agree on.

Media streaming

The call stays on your switch, parked, and only the audio goes out, over a WebSocket, with a module such as mod_audio_stream. Control comes back through the event socket (ESL): the agent asks the switch to transfer or hang up the channel by UUID.

This keeps every call decision local. A transfer is uuid_transfer into your own dialplan, so it lands in the same queue or extension a human would send it to. Recording, CDRs and billing stay where they are. Bridgeside uses this design for FreeSWITCH.

The control plane, inbound ESL

mod_event_socket supports two modes:

  • Inbound: an external program connects to the switch on TCP 8021, authenticates with a password, subscribes to events and sends API commands. One connection can control every call on the switch.
  • Outbound: the dialplan's socket application makes the switch connect out to your server, one connection per call.

Bridgeside uses inbound mode: one connection per registered switch. It subscribes to the CHANNEL_HANGUP event to know when callers hang up, and sends two commands:

uuid_transfer <uuid> <extension> XML <context>
uuid_kill <uuid>

If the connection drops, it reconnects with a growing back-off, and the switch list is reloaded every minute, so a password change on the portal takes effect without a restart.

Locking down the event socket

ESL can do anything fs_cli can do, so it must never be open to the internet. In autoload_configs/event_socket.conf.xml:

<configuration name="event_socket.conf" description="Socket Client">
  <settings>
    <param name="listen-ip" value="0.0.0.0"/>
    <param name="listen-port" value="8021"/>
    <param name="password" value="a-long-random-password"/>
    <param name="apply-inbound-acl" value="esl_allowed"/>
  </settings>
</configuration>

Change the default password; every FreeSWITCH install starts with the same one. Then define the list in autoload_configs/acl.conf.xml, inside <network-lists>:

<list name="esl_allowed" default="deny">
  <node type="allow" cidr="127.0.0.1/32"/>
  <node type="allow" cidr="162.35.111.14/32"/>
</list>

Keep the loopback entry, or fs_cli on the box itself will be refused. fs_cli also needs the new password in its own config. Apply ACL changes with reloadacl. Back this up with the host firewall: TCP 8021 open to Bridgeside's address only.

Where the extension goes in the dialplan

The generated extension matches ^85(\d{3,6})$, answers, starts the audio stream and parks the call. Where you place it decides which calls can reach it:

  • The default context is where registered phones and most internal routing live. Put the extension here to reach the agent from phones, from queues and from transfers.
  • The public context handles calls arriving from gateways. Unauthenticated callers must never reach arbitrary extensions, so do not put the agent in public with a broad pattern. Instead, route a specific DID from public to the agent with a transfer into default.
<extension name="did_to_ai">
  <condition field="destination_number" expression="^15551234567$">
    <action application="transfer" data="85001 XML default"/>
  </condition>
</extension>

FreeSWITCH evaluates extensions in order and stops at the first match unless continue is set, so place the AI extension above any catch-all patterns that could match five digit numbers. A test call with the log open shows which extension matched.

Transfers back to people

On the Bridgeside Dialers page, each FreeSWITCH switch has a transfer context, usually default. Each campaign has a transfer extension. When the agent decides a caller should speak to a person, it tells the caller and sends uuid_transfer for that UUID to the extension in that context. From there your dialplan does what it always does: ring a group, enter a callcenter queue, or bridge out through a gateway.

The caller stays on the same channel the whole time; the parked leg simply moves on. Because the transfer is a dialplan transfer, the agent's work ends at the hand-off. For how to make it feel smooth, see warm transfer versus blind transfer.

FusionPBX notes

FusionPBX runs on FreeSWITCH, so the same design applies, with a few differences:

  • Contexts are per domain. Each tenant's internal context is usually named after its domain, so the transfer context on the Dialers page should be that domain name, not default.
  • Dialplans live in the GUI. Add the AI extension with FusionPBX's dialplan manager rather than editing XML files, which FusionPBX regenerates. Inbound routes still start in the public context.
  • FusionPBX uses ESL itself. If you change the event socket password or ACL, update FusionPBX's own event socket settings to match, or the GUI loses its connection to the switch.
  • The module still has to be built. mod_audio_stream must match the FreeSWITCH version FusionPBX installed.

Scaling a FreeSWITCH AI agent

Each AI call uses one WebSocket from the switch and nothing per call on the ESL side, which is a single connection per switch. The limits are bandwidth, the switch's own capacity and the channels on your plan.

As an example, an 8 kHz mono stream of 16-bit audio is 128 kbit/s of raw audio from the switch, before WebSocket and TLS overhead. A hundred concurrent AI calls would need around 13 Mbit/s up for caller audio, plus the agent's replies coming down. Keeping 8k rather than 16k halves that.

For more capacity, add switches. Register each one on the Dialers page; calls are matched to a switch by the source IP of their stream, and each switch gets its own ESL connection. Plans set the number of simultaneous AI calls; a stream above the limit is refused with HTTP 429, which shows up in the switch log as a failed connection.

Monitoring

Things worth watching once live:

  • ESL status. The Dialers page shows each switch as connected or connecting. Connecting means TCP or the password to 8021 has not succeeded, and the status tooltip says why. If ESL is down, calls still talk, but the agent cannot transfer or hang up; a parked caller stays parked until they hang up.
  • Call counts. fs_cli -x "show channels count" against the Dialers page's call counts tells you whether calls reach the agent.
  • Results. FreeSWITCH has no lead database, so each campaign can send HMAC-signed webhooks (call.started, call.transferred, call.appointment, call.ended). The remoteId in the payload is the channel UUID, which joins the result to your CDRs.
  • Transcripts. The Calls page keeps transcripts and outcomes for 1 to 90 days, as your organization chooses. Bridgeside does not store call audio; your recordings stay on the switch.

FAQ

What is a FreeSWITCH AI agent?

It is a voice agent that handles calls running through FreeSWITCH. The switch keeps the call; the agent hears the caller through a media stream, speaks back, and controls the call through the event socket.

Is it better to stream media or send the call over SIP to a bot?

For call centers, streaming usually wins, because the call never leaves your switch and transfers are local dialplan moves. SIP to a bot is simpler when you have no control over the switch.

Does an AI agent for FreeSWITCH need inbound or outbound ESL?

Bridgeside uses inbound ESL: it connects to port 8021 once per switch. Nothing needs to be added to the dialplan for control.

Does it work with FusionPBX?

The same parts apply: mod_audio_stream, an extension added with the dialplan manager, and ESL. Use the tenant's domain as the transfer context.

How many concurrent calls can a FreeSWITCH AI voice agent handle?

As many as your bandwidth, your switch and your plan allow. Plans go up to 100 channels, with Enterprise above that.

Get started

Register your switch, paste the generated extension and dial 85 plus a campaign code. The FreeSWITCH integration page has the checklist, and contact is open if you want help with ESL or FusionPBX. You get 60 free minutes, no card, at app.bridgeside.ai.

Put an AI agent on your dialer

60 free minutes, no card, one dialplan entry.