Build Low-Latency Voice AI with Quillman on Modal

Speaking to most conversational AI agents feels uncannily like talking on a satellite phone with a four-second delay whilst trapped in an awkward lift. You finish a sentence, stare into the middle distance, wonder if your Wi-Fi has died, and then—just as you open your mouth to ask if it heard you—the machine talks over you.

The open-source repository modal-labs/quillman sets out to banish that conversational cringe. Maintained by the team at Modal Labs, Quillman is an end-to-end voice chat architecture that demonstrates how to stitch together automatic speech recognition (ASR), large language model (LLM) generation, and text-to-speech (TTS) into a streaming pipeline with sub-second time-to-first-audio.

+------------------------------------------------------------------------+
|                          Quillman Audio Loop                           |
|                                                                        |
|  [User Mic] ---> (Silero VAD) ---> [Whisper ASR]                      |
|                                         |                              |
|                                         v (Text Stream)                |
|  [Speaker]  <--- [Chunked TTS]  <--- [vLLM / LLM]                     |
|                   (Streaming)                                          |
+------------------------------------------------------------------------+

---

What Is Quillman?

Direct Answer: Quillman is a reference implementation of a real-time, bidirectional voice agent built on Modal's serverless GPU infrastructure. It uses WebSockets to pipe user microphone input through voice activity detection (VAD) and Whisper transcription, streams tokens from an open-weights LLM running on vLLM, and synthesises speech concurrently using a streaming TTS engine.

Rather than waiting for every sentence to finish generating before producing audio, Quillman breaks down the pipeline into overlapping streams. As soon as the LLM yields its first handful of tokens, the TTS model begins processing phonemes and pushing audio buffers back down the socket.

---

The Latency Math: Why Voice Pipelines Stumble

In a naive "serial" voice pipeline, latency compounds aggressively:

1. Audio Recording & VAD: 300ms to detect silence.
2. Transcription (ASR): 400ms to process the full utterance via Whisper.
3. LLM Generation: 1,200ms to generate a complete answer.
4. Speech Synthesis (TTS): 800ms to convert the entire string into a WAV file.
5. Total Lag: Nearly 3 seconds before the user hears a single peep.

Quillman decouples these stages. By employing chunked sentence splitting on the incoming token stream, speech synthesis runs in parallel with LLM decoding. The user hears the start of the machine's answer while the LLM is still drafting the middle of it.

Metric / Attribute	Naive REST Pipeline	Quillman Streaming Pipeline
Transport	Standard HTTP POST calls	Bidirectional WebSockets
Turnaround Latency	2,500ms – 4,500ms	400ms – 900ms
TTS Execution	Batch across full response	Pipelined per clause / punctuation
Infrastructure	Always-on dedicated instances	Auto-scaling serverless GPUs (Modal)
Cold Starts	Zero (expensive) or High	Container snapshotting / warm pooling

---

Key Architectural Components

1. Silero VAD & OpenAI Whisper: Audio chunks hit the server over WebSockets. Silero VAD monitors incoming frames to spot when speech stops, instantly passing the cached buffer to an optimised Whisper instance.
2. vLLM Engine: For the conversational core, Quillman plugs into vLLM running modern open-weights models (such as Mistral or Llama). It begins yielding tokens immediately via continuous batching.
3. Pipelined TTS Worker: Instead of feeding full paragraphs, a custom boundary parser grabs complete phrases (splitting cleanly on commas, semicolons, and full stops) and feeds them directly into models like Bark, XTTS, or Kokoro.
4. Modal Serverless Orchestration: The heavy lifting runs on remote GPUs provisioned on-demand, handling scaling and container pooling without requiring you to nurse a cluster of idle A100s.

---

Getting Started: Deployment & Usage

Because Quillman relies on Modal's Python-native cloud primitives, you do not need to wrestle with Kubernetes manifests or messy Docker networking just to test the pipeline.

1. Prerequisites and Setup

Install the Modal client and authenticate with your account:

pip install modal
modal setup

Clone the Quillman repository:

git clone https://github.com/modal-labs/quillman.git
cd quillman

2. Deploying the Backend

Quillman packages its components into Modal functions decorated with specific GPU requirements. You can stand up the entire backend with a single command:

modal deploy src.app

Modal builds the container images in the cloud, pulls the necessary model weights (Whisper, LLM checkpoints, and TTS voice profiles), and mounts the WebSocket endpoints.

3. Running the Local Client

Once deployed, Modal gives you a web URL hosting the FastAPI backend and static frontend. You can also run the local development client to inspect pipeline latency:

import asyncio
import websockets
import json

async def talk_to_quillman(server_url: str):
    async with websockets.connect(server_url) as ws:
        # Handshake with chosen voice and model parameters
        await ws.send(json.dumps({
            "type": "config",
            "model": "meta-llama/Llama-3-8B-Instruct",
            "voice": "af_bella"
        }))
        
        print("Connected. Streaming voice audio...")
        # Listening loop receives binary audio chunks while sending mic input
        async for message in ws:
            if isinstance(message, bytes):
                # Play raw PCM audio frame directly through your local output
                pass
            else:
                event = json.loads(message)
                if event.get("type") == "transcript":
                    print(f"Heard: {event['text']}")

# asyncio.run(talk_to_quillman("wss://your-modal-app.modal.run/ws"))

---

Why It Stands Out

Developer consensus on YouTube technical breakdowns and community channels usually points to one harsh reality: building voice agents is rarely about the models themselves; it is about buffer management and latency budgets.

Quillman stands out because it provides an unpretentious, transparent blueprint for solving the streaming pipeline problem. It does not hide the mechanics behind an opaque SaaS abstraction or lock you into proprietary speech APIs. If you want to swap Whisper for a fine-tuned Whisper-large-v3-turbo, swap the LLM for a local Qwen variant, or drop in a newer low-latency TTS architecture, the seams are visible and accessible. It gives power users and builders complete control over their latency stack without running up astronomical hosting bills for idle GPU hours.