Nobody Prompt Caches Their Voice Agent

The First Second Is The Whole Product

In voice, latency isn't a metric. It's the product. First impressions are everything, and a voice agent gets about one second to make one.

Think about it. A text chatbot can take three seconds and nobody cares, you're just staring at a text box. But a voice agent that takes three seconds to answer? Feels broken. I've literally watched people start going “hello?” at the two-second mark. Humans start responding like 200ms after a turn ends, and every 100ms past that, the caller's wondering if the call dropped.

I got our first response from around 3.4 seconds down to under a second. And the thing that did it lives somewhere almost nobody in voice AI is even looking: prompt caching. This piece is about the blind spot, and what I built once I saw it.

The Prefill Problem

A production voice agent doesn't run on some cute little system prompt. The one I built [Boardy] carries the persona, the meeting context, the participant roster, the session brief, plus every tool schema the session is allowed to call. Tens of thousands of tokens before the user has said a single word.

So here's the problem. The first LLM call of a session has to chew through that entire prompt before it can generate its first token. That part is called prefill. The pipeline I built still opened with the caller sitting through a full prefill of the biggest prompt in the system. Even on the fastest LLM I could get my hands on, Gemma 4 31B running optimized on LiveKit Inference, the first response still took three to four full seconds.

As I'm writing this, you can kinda think of first-turn latency like this:

T_{\text{first}} = T_{\text{prefill}}(P) + T_{\text{decode}}

In normal words: $T_{\text{first}}$ is just the time from the user finishing talking to the moment they hear audio back. Two parts. $T_{\text{prefill}}(P)$ is the model going through the whole system prompt $P$ before it can say anything. $T_{\text{decode}}$ is spitting out the first few tokens. Decode is nothing. Prefill is where all the time goes.

And this is what I noticed: the prompt barely changes between turns. The persona, the tools, the context. It's basically all the same prefix every time. Which means the prefill you're paying on turn one is work you could've done before the conversation even started.

cold

ringback

prefill, full prompt

first audio

~3.4s

warmed

ringback + warmup

cache hit

first audio

~0.9s

ringback

prefill, full prompt

ringback + warmup (2 synthetic calls)

cache hit / first audio

First-response latency with and without warmup.

The Demo Trap

Part of why nobody notices: the demos hide the problem. Go play with any voice demo online. LiveKit's demo page, Grok voice mode, whatever. They're fast. And they'll make you feel like you're doing something wrong when your production agent takes two seconds to say hello.

The trick nobody mentions: their system prompt is tiny.

A demo prompt is like, a persona line and maybe a tool or two. A few hundred tokens. Prefill on a few hundred tokens is nothing, so of course the first token flies out. I haven't seen their exact prompts, but I know what's not in there: thirty thousand tokens of tool schemas, participant rosters, and business context.

This is also why I'd get those random half-joking complaints about why ours was slower than the one hosted on some webpage. That one has like twelve tokens to think about. Ours has thirty thousand.

Prefill time scales with prompt length. So demo numbers stop meaning anything the second you give an agent real capabilities. Real tools, real context, real instructions. The prompt gets big, prefill gets slow, and no faster model fixes that, because it's structural.

You can't demo your way out of prefill. You gotta engineer around it.

Go read everything written about voice AI latency in the last two years. It's all TTS streaming, VAD tuning, barge-in handling, turn detection, faster models. All real problems. But nobody talks about the prompt. Like, at all.

I went looking for prior work on caching for voice agents and found nothing. Zero. The actual research on prompt caching, like this paper, is all coding agents and deep-research agents. Voice doesn't show up once. The field where latency matters most is just not in the conversation. A faster first token is a footnote for a research agent. For a voice agent it's the difference between “hello?” and a conversation.

Caching pays off when two things are true: the prompt is big, and latency matters. Voice is the extreme of both, the biggest prompts and the tightest latency budget of any interface. A text agent takes three seconds and nobody blinks. A voice agent doesn't get that luxury. And the technique that helps most is the one nobody's using. That's the blind spot.

There's also a split happening in voice AI that decides who actually needs this. The workflow bots, receptionists, appointment bookers, dispatch callers, run tiny prompts. Prefill is a rounding error for them, and they can ignore this whole piece. But everything's heading toward consumer assistants. Muse's voice mode, Grok's voice mode, ChatGPT's voice mode. Agents that can do anything, which means they carry everything. Dozens of tools, full user context, long instructions. Their prompts look like ours.

One more thing: everything I've described is the cascaded world. Speech-to-text into an LLM into text-to-speech, which is what I built and what basically everyone ships in production. Speech-to-speech models are the other bet, a single model doing it all end to end, and their latency story is completely different. There's no prompt prefill to cache when the whole thing is one audio stream. But cascaded is what's actually running today, and in cascaded systems, prefill is the bottleneck.

How Prompt Caching Actually Works

Honestly, when I started writing this article, I had no idea how the KV cache actually worked. I knew the words, prefill, decode, cache hit, but not what was actually happening under the hood. Then I read this paper and it just clicked.

So a transformer doesn't just read your prompt and start talking. During prefill, it goes through every token and computes a set of key and value vectors (that's the KV cache) that capture what each token means in context. Then when it's generating, it doesn't redo all of that. It reuses the cached keys and values and only does the math for the new token. Once I saw it explained that simply, the whole thing made sense. Prefill's entire job is building that table. So “warming the cache” is literally just doing that computation before anyone's waiting on it.

Prompt caching is the provider keeping that table around between requests. If your next request starts with the same tokens (same system prompt, same instructions), the provider skips rebuilding the KV cache for the shared prefix and jumps straight to decoding. Prefill on tens of thousands of tokens becomes prefill on basically zero.

And the paper's biggest finding is literally what I built: caching the full conversation can actually make things slower, because dynamic content keeps invalidating the cache. Caching just the stable system prompt is what wins. That's exactly where my warmup draws the line. Stable prefix gets cached, instructions and schemas, nothing dynamic. In voice you don't have room for a cache strategy that sometimes backfires.

The TTL Problem

Okay, there's a catch. And it's the reason voice can't just copy what text agents do.

Caches expire. Anthropic documents five minutes of inactivity before the cache evaporates (every hit resets the clock, with a one-hour option). OpenAI doesn't publish a number, but their automatic cache works the same way: minutes idle, then it's gone. Either way, the design assumes something voice doesn't have: constant traffic.

A busy text endpoint keeps its cache hot for free. Thousands of requests a minute, the five-minute window never closes. But voice calls are sparse. A call now, the next one twenty minutes later, maybe an hour. By the time the next caller dials in, the cache from the last call is long dead. If you're relying on the provider's cache surviving between sessions, you're getting nothing.

So the way to think about it: for voice, you can't inherit a warm cache. You have to build one, every single session. The question is how, when the caller's already on the line and the clock is running.

Nobody prompt-caches their voice agent, and the TTL is the quiet reason why. The cache was designed for traffic patterns voice doesn't have. So you need an adapter, something that builds the cache inside the session instead of waiting to inherit one. That's what I built.

The Trick: Two Synthetic Calls

The fix I landed on is almost embarrassingly simple. Most AI engineers I mention this to are surprised when they find out it reduces latency.

Before the session starts, I just make two LLM calls carrying the exact prompt the live session's gonna use, and throw the answers away. Each one carries the full agent instructions as the system message, the user context, and the session's real tool schemas. Then a dummy user message that's basically “don't do the actual task, just reply OK.” Output capped at 8 tokens. One shared 12-second deadline across both calls, no retries. I drain the stream chunk by chunk so the provider commits the shared prefix into its KV cache, and then the response gets tossed. Nothing is published. The caller has no idea it happened.

assemble prompt

full instructions, user context, tool schemas

fire 2 synthetic calls

max 8 output tokens each, one shared 12s deadline

drain streams

KV cache committed, the “OK” responses discarded

guard tools

a cache operation, not an agent

The warmup as a pipeline. It runs during ringback, before the session starts.

I was messing around with where to put this, and honestly the placement mattered as much as the trick itself. On phone calls it runs while the caller's hearing ringback, in parallel with caller-ID setup. Dead air I'd be wasting anyway becomes the warmup window.

One thing I got wrong the first time and had to fix: if the model tries to call a tool during warmup, execution has to throw. The warmup is a cache operation, not an agent. Tool calls come back, get drained, never run.

It Gets Better Every Turn

The warmup is just the first domino. The part I didn't appreciate until I watched it run: the cache doesn't just stay warm, it compounds.

Turn one is a cache hit on the system prompt. The provider serves the whole prompt from cache and only computes the new stuff, the user's first message. Then it folds those new tokens into the cache too. Turn two is a cache hit on the system prompt plus turn one. Turn three adds turn two. Every turn, the cached prefix keeps growing, and the only tokens you ever pay prefill on are the delta: just what changed since last turn.

The math is kinda sick. Say the system prompt is 30,000 tokens and each turn adds about 300. Turn one serves 99% from cache. Turn ten serves 99.4%. The uncached sliver stays tiny no matter how long the call goes.

Compare that to no caching. Without it, every turn recomputes the entire conversation from scratch. Turn ten pays prefill on 33,000 tokens. Turn fifty pays on 45,000. Long sessions don't just start slow, they get slower. Caching flips that completely: per-turn prefill stays flat at the delta, forever.

One caveat, same one from the paper: this only works while the prefix stays byte-identical. Change anything in the middle of the prompt and everything after it has to be recomputed. Keep the stable stuff up front and the cache compounds on it.

The Economics

So what does it cost? Two requests per session. Each one sends the entire system prompt as input and generates at most 8 tokens back. That's it. That's the whole cost of the warmup: two full-context prefills per session.

Honestly speaking, this probably adds up in the long term, but you're buying someone's first impression, and I strongly believe any amount is worth this.

What you get: every turn after that starts from cached tokens.

\text{cost} = 2 \cdot C_{\text{prefill}}(P) \quad\text{vs.}\quad \text{savings} \approx N \cdot C_{\text{prefill}}(P_{\text{shared}})

For a session with $N$ turns, you're trading two prefills upfront for $N$ turns that skip prefill on the shared prefix. It pays for itself the second the session goes more than a couple turns. And voice sessions always do.

I'm not gonna quote dollars here. That depends on inference pricing for the pinned model, and I'd rather show the token math than a number that goes stale.

Without Warmup	With Warmup
First turn pays full prefill	Prefill moves to ringback dead air
Demo numbers don't transfer	Production latency matches the math
Random slow turns	2× prefill per session, accounted

TLDR

I took our first response from around 3.4 seconds to under a second by doing the thing almost nobody in voice AI is doing: actually taking prompt caching seriously. Voice agents have the biggest prompts and the tightest latency budget, so caching matters more here than anywhere. But all anyone talks about is TTS and turn-taking, the research skips voice completely, and demos hide the problem with tiny prompts. Two synthetic 8-token calls during ringback move prefill out of the critical path, because the KV cache only has to get built once. And it compounds: every turn grows the cached prefix, so per-turn prefill stays flat at the delta no matter how long the call runs. As voice AI moves from workflow bots to real assistants, this stops being an optimization and becomes the whole approach.