Everyone talks about "AI inference" like it's one problem. It isn't. Serving text and serving speech are as different as catering a wedding and running a dosa stall. Once I saw why, half of the weird engineering pain I kept running into with voice AI stopped feeling weird.
Here's how I think about it now.
Part 1: Text generation is a planning problem
An LLM request has two phases, and they behave nothing alike:
- Prefill: the model reads your whole prompt at once. Compute-bound. GPU goes brrr.
- Decode: the model writes one token, then another, then another. Memory-bound. Every single token means re-reading the entire KV cache from memory.
That asymmetry is the reason serving engines exist. Everything (batching, caching, scheduling) is downstream of "prefill is fast and parallel, decode is slow and sequential."
The naive way to batch fails immediately. Wait for N requests, run them together, return all results, and now your 3-word request is stuck behind someone's essay. Continuous batching fixes this: requests hop in and out of the running batch token-by-token, so nobody waits on the slowest guest in the room.
Part 2: The KV cache, three ways to remember
Every token needs its key/value vectors held in memory so future tokens can attend back to it. How you store that memory is basically the whole ballgame.
The old way reserved a banquet hall for one guest: memory sized for the worst case, max context length, every time. Ordered a quick coffee? Doesn't matter, you got the entire hall. Wildly wasteful, and it capped how many concurrent users you could even serve.
vLLM is the thali mess with steel containers. PagedAttention splits the KV cache into fixed-size blocks, like pre-portioning food into standard containers instead of cooking one giant thali per customer. If two orders are identical from the very first bite, same starting sequence, byte for byte, the mess hall hands out the same container to both. That's prefix caching. But it only works if the match is exact and starts at position zero.
SGLang is the wedding caterer. Picture a 500-person shaadi where nobody cooks per-plate. The caterer builds a shared prep tree, common base first, then branches for each dietary preference. RadixAttention works exactly like this: a radix tree of KV cache shared across every request currently being served, not just sequential repeats. Two totally different guests who both want "spicy Punjabi" share that branch automatically. That's the real upgrade over vLLM: vLLM recognizes a repeat only if it's the exact same words as the last customer, SGLang recognizes what's shared across the entire hall, live.
TensorRT-LLM is the caterer who pre-plans the entire menu in advance, compiling the whole kitchen workflow ahead of time for a fixed menu and guest count. Faster once running, but change the guest list and you're rebuilding the plan.
| Engine | Caching style | Best for |
|---|---|---|
| Naive | Reserve the whole hall per guest | Nobody, anymore |
| vLLM | Steel containers, exact-match reuse | General purpose, repeat traffic |
| SGLang | Shared prep tree across the whole hall | Many concurrent sessions with shared structure |
| TensorRT-LLM | Pre-planned kitchen, compiled ahead of time | Max steady-state throughput, fixed shapes |
A few published numbers that back this up: PagedAttention gets 2-4x throughput over pre-vLLM serving (FasterTransformer/Orca) at the same latency, and up to ~24x over naive HuggingFace Transformers serving. SGLang's RadixAttention gets ~37% lower p50 TTFT and ~41% lower p95 TTFT than vLLM when more than 60% of requests share a prefix; on unique prompts the two land within ~5% of each other.
Part 3: Speech isn't catering. It's a dosa stall.
Everything above assumes you know the full order in advance. You can tokenize a prompt, see its whole shape, plan around it. Speech doesn't work that way. There's no guest list. Orders are shouted, mid-crowd, in real time, and you're the one dosawala on shift.
- VAD is your ear filtering the crowd. Car horns, other stalls, background chatter, none of it is for you. That's admission control, a totally different problem from "which request runs next."
- Streaming STT is you guessing mid-sentence. "Plain dosa..." No wait. "Plain dosa with extra chutney." You're constantly revising your best guess as more sound arrives. LLM decode never does this: a generated token is final the moment it's out. A transcript changes its mind.
- TTS is the tawa. You can't batch-cook ten dosas the way a mess hall preps ten identical thalis in advance. Every dosa has to land hot, in real time. Cook too many at once and the first ones go cold, that's blowing your real-time factor.
- There's no shared prep tree. The caterer can pre-cook a common base for the whole wedding. The dosawala can't pre-fry a dosa and hand out copies. That's exactly why nothing like RadixAttention exists for raw audio yet.
Silero VAD backs this up on the compute side: it processes a ~32ms audio chunk in under 1ms of CPU time. The math is nearly free, the 85-100ms of latency you actually feel is the buffering window, not the model. On the TTS side, time-to-first-audio is what turn-taking latency really depends on, not RTF: Kokoro delivers ~28-45ms first-chunk latency on an RTX 5090, XTTS-v2 takes ~320ms. Both are "real-time capable" by RTF, but the user waiting on first audio feels the 320ms, not the ratio.
Part 4: Where speech-to-speech is trying to go
The industry's answer to "no shared prep tree for audio" is speech-to-speech models, Moshi, GPT-4o Realtime, Gemini Live. The trick: turn audio into discrete, token-like codes so a single model can consume and emit sound directly, skipping the STT to LLM to TTS relay entirely. In dosa-stall terms, it's an attempt to pre-batter and pre-cache the dosa after all, by making sound behave enough like text tokens that the caterer's tricks finally apply. The tradeoff: collapse three engines into one model and you lose the ability to independently swap or tune each stage.
Most voice pipelines today are still three separate engines duct-taped into one relay, and none of them share a request lifecycle. That mismatch is where most of the real engineering pain lives:
- Cross-engine metric attribution: one conversational turn touches three engines, each with its own idea of "done." Get the turn-identity tracking wrong and your latency dashboards quietly lie to you.
- No shared caching across the relay: RadixAttention helps the LLM stage, nothing for STT or TTS.
- Interruption handling spans all three state machines at once: VAD, generation state, and playback state all have to agree, fast.
Part 5: What breaks specifically under concurrent load
Everything above holds for one conversation. The moment you have many concurrent streaming sessions on shared infra, a different class of problem shows up, not "is the pipeline correct" but "does it stay correct under contention."
- GPU batch fairness across sessions. A burst of new sessions can push an in-flight session's TTS chunk past its real-time deadline, even though no single request was slow on its own.
- Head-of-line blocking at the STT stage. STT is usually smaller and cheaper than the LLM, so teams under-provision it. When STT saturates first, LLM and TTS sit idle, the bottleneck moves to the cheapest engine in the relay.
- VAD false-positives multiply with concurrency. The rate of spurious activations scales with concurrent sessions, not per-session load. A VAD tuned fine at 10 concurrent calls can start admission-flooding the STT stage at 200.
- KV cache eviction pressure couples unrelated conversations. A burst of new, unrelated sessions can evict a long-running conversation's cached prefix mid-turn, so that user's next token silently gets slower.
- Backpressure has no single valve. Shed load at the VAD, the LLM, or the TTS, and each choice produces a different, user-visible failure mode.
- Session affinity fights autoscaling. The relay shares mutable state across one conversation, so you can't load-balance mid-conversation like stateless HTTP. Scale-down has to wait out every long-running session.
Part 6: A couple of numbers from poking at local TTS stacks
Quick concurrency sweeps on a couple of local TTS setups, out of curiosity. They lined up with Part 5 closer than expected.
NeuTTS on an H100 (concurrency 1/2/4/8/16):
| Concurrency | p50 Latency | Mean RTF | Throughput | Peak VRAM |
|---|---|---|---|---|
| 1 | 1.36s | 0.171x | 5.82x | 3.7 GB |
| 2 | 2.36s | 0.288x | 6.63x | 6.5 GB |
| 4 | 3.74s | 0.463x | 8.39x (peak) | 12.9 GB |
| 8 | 10.23s | 1.298x | 5.94x | 25.6 GB |
| 16 | 25.00s | 3.085x | 5.02x | 51.1 GB |
Throughput peaks at concurrency 4, then drops, while p50 latency goes from 1.36s to 25s.
Latency breakdown: backbone LLM generation is 79% of total time, codec decoding 18.6%, everything else under 3%.
PocketTTS, serialized mode, CPU vs. H100:
| Concurrency | GPU Throughput (chars/s) | GPU RTF | CPU Throughput (chars/s) | CPU RTF |
|---|---|---|---|---|
| 1 | 89.4 | 5.03x | 51.2 | 2.45x |
| 2 | 94.4 | 5.06x | 44.5 | 2.38x |
| 4 | 94.5 | 5.00x | 46.2 | 2.40x |
| 8 | 98.2 | 5.00x | 46.1 | 2.39x |
RTF and throughput barely move with concurrency on either device.
Per-voice CPU breakdown (8 voices, same sentence): RTF 2.44x-2.53x, TTFT 135-146ms across the board.
Takeaway
Text serving got two years of fast, focused engine competition, vLLM, then RadixAttention, then TensorRT-LLM, because the workload was uniform enough to iterate on shared abstractions. Speech serving is still three separate stalls sharing one street, because the I/O shape is different at every stage, and the thing that would actually unify them, S2S, is only now maturing enough to borrow the caterer's tricks.
References: vLLM/PagedAttention paper (SOSP 2023), vLLM blog, SGLang project, Silero VAD performance metrics, Kokoro vs XTTS-v2 latency comparison.