AssemblyAI details Python voice agent build with tool calling

News

AssemblyAI published a Python tutorial for a real-time voice agent that combines streaming speech recognition, LLM Gateway and text-to-speech.

AssemblyAI has published a tutorial showing how to build a real-time voice agent in Python that listens, reasons, calls functions and speaks back using three files and one API key. The guide centers on tool calling, and it uses Universal-3.6 Pro Realtime for streaming speech-to-text, an LLM through LLM Gateway, and the AssemblyAI Text-to-Speech API for spoken replies, according to the company’s Oct. 29, 2026 post.

Evidence and context

The tutorial says the agent can listen through a microphone, decide when to invoke Python functions, speak the result and stop speaking when interrupted. AssemblyAI frames that as a production-style loop in which the script owns the orchestration: microphone audio goes to Universal-3.6 Pro Realtime, turn events trigger an LLM request with tools, tool results are fed back into the model, and the final response is streamed to text-to-speech.

AssemblyAI argues that the listening layer is a key differentiator for voice agents. In the company’s English voice-agent benchmark, which it describes as 12,460 scripted caller scenarios, Universal-3.6 Pro Realtime posted a 5.19% word error rate versus 7.78% for ElevenLabs Scribe v2, 8.64% for Deepgram Nova-3 and 13.50% for Deepgram Flux EN. For structured values, AssemblyAI says the model recorded a 14.4% entity error rate on the same benchmark, compared with 30.1% for Deepgram Flux EN. These are vendor-reported results from AssemblyAI, not independently verified measurements in this material.

The post also highlights agent_context, a parameter that lets the speech-to-text model ingest the agent’s own last line. AssemblyAI says that matters for short replies such as emails, account numbers and confirmations, where context can help resolve ambiguous speech. The tutorial shows updating agent_context after each reply so the next utterance is transcribed with the previous answer in mind.

Tool calling, barge-in and deployment details

AssemblyAI’s example defines tools as OpenAI-style JSON schemas and passes them through LLM Gateway. The guide says the code reads all returned choices, parses JSON arguments, runs the matching Python function and sends the result back as a tool message until the model answers in plain text. The sample tools include weather lookup, memory storage and recall, but AssemblyAI notes that any Python function can be wired into the loop.

The tutorial also addresses interruption handling. It uses turn events from the speech-to-text stream to detect when a user is speaking over the agent, filters out backchannels such as “mhm” and “yeah,” and cancels text-to-speech mid-chunk when a real interruption occurs. AssemblyAI says that approach lets the agent stop within one audio chunk instead of finishing a sentence after the user has already cut in.

For deployment, the post says the local microphone setup can be adapted for telephony or web apps. It mentions Twilio Media Streams for phone calls, with 8 kHz μ-law audio accepted directly by both services, and says browser-based audio should keep the API key on the server. The article also notes that text-to-speech sessions can last up to an hour and close after two idle minutes, which is why the example sends keep-alive calls during inactivity.

AssemblyAI closes by comparing the cascaded approach in the tutorial with a single speech-to-speech model. The company says the split design preserves a visible transcript, keeps each layer replaceable and improves handling of structured data. The timing of any next product update or release beyond this tutorial was not supplied in the source material.