What 100ms Actually Means

When we tell people our target for end-to-end avatar response time is under 200 milliseconds, the most common reaction is a polite nod followed by "is that hard?" Yes. It is extremely hard. Here's why, and what we've learned about where latency hides.

Why latency matters more than you think

There's a common assumption that latency is a nice-to-have, that users will tolerate a bit of delay for better quality. In many contexts, that's true. Nobody minds waiting three seconds for a better image generation. But conversation is a real-time protocol. Humans take turns speaking with that precision because their brains run a timing mechanism that predicts when the other person will finish and pre-plans a response, not because it's polite.

Break that timing and you break the conversation.

Where the milliseconds go

Let me walk through a typical AI avatar pipeline and show where time is actually spent. This is based on our production system at Anam, though the general shape applies to any real-time AI interaction.

Add those up and you get 260ms. That's already over our 200ms target, and this is the optimistic scenario: no queuing, no cold starts, no network jitter. In production, you need headroom for variance. If your P50 is 200ms, your P95 is probably 400ms, and your P99 is somewhere that makes you uncomfortable.

The challenge is optimising all of the components at once without tanking quality.

Component by component

Audio capture and processing (20ms)

The browser captures audio in chunks. The Web Audio API typically works with buffer sizes of 256 to 4096 samples. At 48kHz, a 1024-sample buffer gives you about 21ms of audio per chunk. You can go smaller, but smaller buffers increase CPU load and risk audio glitches.

const audioContext = new AudioContext({ sampleRate: 48000 });
const processor = audioContext.createScriptProcessor(1024, 1, 1);
 
// Each callback gives us ~21ms of audio
processor.onaudioprocess = (event) => {
  const inputData = event.inputBuffer.getChannelData(0);
  // Send to STT pipeline
  sendToTranscription(inputData);
};

There's also the voice activity detection (VAD) step: determining when the user has actually finished speaking. Get this wrong and you either cut them off mid-sentence or add hundreds of milliseconds waiting for silence that's actually a pause.

Speech-to-text (40ms)

Streaming STT has improved a lot recently. The trick is using a model that returns partial transcriptions with low enough latency that you can begin inference before the user has finished speaking. Think of it as speculative execution for conversation: you start processing the likely input before it's confirmed.

interface TranscriptionEvent {
  text: string;
  isFinal: boolean;
  confidence: number;
  latencyMs: number;
}
 
// Start LLM inference on high-confidence partials
function onTranscription(event: TranscriptionEvent) {
  if (event.isFinal || event.confidence > 0.85) {
    startInference(event.text);
  }
}

The risk is obvious: if the partial transcription is wrong, you've wasted compute on an incorrect input. In practice, high-confidence partials are correct often enough that the average latency saving outweighs the occasional wasted inference.

LLM inference (80ms)

This is usually the biggest single component, and the hardest to optimise because you don't control the model. If you're using a hosted API, you're subject to their queuing, their hardware, and their cold start behaviour.

A few things that help:

Streaming responses. Don't wait for the full response. Start TTS on the first sentence while the rest is still generating. Should be obvious, but I still see implementations that wait for the complete response.

Prompt engineering for latency. Shorter system prompts mean fewer tokens to process. This sounds obvious but I've seen system prompts that are two thousand tokens long because someone kept appending instructions. Every token in the prompt adds latency.

Model selection. Smaller models are faster. A well-tuned seven-billion parameter model will beat GPT-4 on latency every time, and for many conversational use cases, the quality difference is negligible. Match the model to the task. At Anam, we support custom models, and one of the hardest ongoing challenges is finding models that are fast enough for real-time use while still producing good conversational output.

// Stream the first chunk to TTS as soon as it arrives
async function streamInference(prompt: string) {
  const stream = await llm.chat.completions.create({
    model: 'our-tuned-model',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
    max_tokens: 150, // Keep responses concise
  });
 
  let buffer = '';
  for await (const chunk of stream) {
    buffer += chunk.choices[0]?.delta?.content ?? '';
 
    // Send to TTS at sentence boundaries
    const sentenceEnd = buffer.match(/[.!?]\s/);
    if (sentenceEnd) {
      const sentence = buffer.slice(0, sentenceEnd.index! + 1);
      buffer = buffer.slice(sentenceEnd.index! + 2);
      await sendToTTS(sentence);
    }
  }
}

Text-to-speech (35ms)

Modern streaming TTS can begin producing audio within 30–50ms of receiving text. You don't need the entire utterance to start generating speech; you just need enough phonetic context to produce the first few frames of audio.

Face generation (45ms)

This is where our specific challenge lies. Generating a photorealistic face that matches the audio, maintains identity consistency, and supports real-time streaming is computationally expensive. We run a custom model that generates face frames at 30fps, which gives us about 33ms per frame.

The trick is pipelining. While frame N is being rendered on the client, frame N+1 is being transmitted, and frame N+2 is being generated. This means the generation latency is amortised across the pipeline rather than added to it directly.

Network round-trip (30ms)

WebRTC gives us the lowest practical latency for browser-to-server communication. But 30ms is optimistic. It assumes the user is close to your edge server and the network is behaving.

In practice, network latency is the component with the highest variance. A user in London connecting to a server in Dublin gets 15ms. A user in Sydney connecting to the same server gets 250ms. That makes multi-region deployment a hard requirement for real-time AI.

Client rendering (10ms)

The final step: taking the generated video stream and rendering it to a canvas element. This is the smallest component but it's not free, especially on mobile devices where GPU resources are shared with the browser's compositor.

The pipeline trick

The numbers above suggest 260ms total, but our actual end-to-end latency is under 200ms. How?

Pipelining. Not every component needs to wait for the previous one to complete. Speech-to-text can begin on partial audio. LLM inference can start on partial transcription. TTS can start on the first sentence. Face generation can begin on the first audio frames.

When you pipeline aggressively, the total latency is closer to the slowest single component plus the overhead of moving data between them, not the sum of all components.

// Simplified pipeline: each stage starts before the previous one completes
async function handleUserSpeech(audioStream: ReadableStream) {
  const transcriptionStream = speechToText(audioStream);         // Starts immediately
  const inferenceStream = streamInference(transcriptionStream);  // Starts on first partial
  const audioOutput = textToSpeech(inferenceStream);             // Starts on first sentence
  const videoOutput = generateFace(audioOutput);                 // Starts on first audio frame
  renderToCanvas(videoOutput);                                   // Starts on first video frame
}

This is the core architectural insight: in real-time AI, the wins come from overlapping the components, not from speeding each one up.

What I'd tell someone building a real-time AI product

Instrument everything. Put timestamps at every pipeline boundary. You can't optimise what you can't measure, and latency problems are almost always in a different place than you expect.

Design for P95, not P50. Your median latency doesn't matter if one in twenty interactions feels broken. Users remember the bad experiences.

Embrace speculation. Start processing before you're certain of the input. The cost of occasionally wasting compute is much lower than the cost of always waiting for certainty.

Deploy multi-region. Physics is not negotiable. The speed of light through fibre is about 200km per millisecond. If your server is 3,000km from your user, that's 30ms of latency that no amount of optimisation can remove. We run GPUs in multiple regions with some product-side logic at the edge to keep things as close to users as possible.

Test on bad networks. Use Chrome's network throttling. Test on 3G. Test with packet loss. Your product needs to work where your users actually are, not where your development machine is.

Every unnecessary millisecond is a small tax on the interaction. Each one is negligible on its own. Together, they're the difference between something that feels alive and something that feels like software.