Nari Labs reports 34 ms p95 first-audio latency for Qwen3-TTS at 10 RPS on one H100
- Nari Labs reports 34 ms p95 audible time-to-first-audio for its Qwen3-TTS 1.7B CustomVoice server at 10 requests per second on one NVIDIA H100 SXM, while maintaining real-time playback.
- In five-minute Poisson open-loop tests, the implementation stayed below 50 ms p95 TTFA through 10 RPS and below 100 ms at 20 RPS; Nari says the four compared alternatives did not reach sub-50 ms p95 TTFA.
- The benchmark measures audible TTFA, reconstructs PCM playback to detect underruns, and uses Deepgram STT on completed audio to check output; each request sends complete text while audio streams back.
- At 10 RPS, Nari reports about 630 characters per second and estimates about $2 per 1 million characters at full utilization of a $4.29-per-hour H100 SXM instance.
- Nari Labs released the Qwen3-TTS implementation and benchmark as open source, and says its WebSocket input-streaming variant also runs below 50 ms while LLM time-to-first-token remains a source of delay.
Hacker News opinions
We built this because TTFA matters for real-time voice apps, and existing open-source stacks were too slow or broke playback when pushed for latency. Our optimized Qwen3-TTS reaches 34 ms p95 TTFA at 10 RPS on one H100, and we published the code and benchmark.
How does this perform on consumer hardware? An H100 result is useful, but I want to know the latency on a desktop GPU.
I liked Nari's Dia models and found them very natural. Has the company shifted from Dia model development toward hosting and serving infrastructure?
Are these cold-start numbers, or are the model and caches already warmed up?
ChatGPT responds quickly by saying things like "hmm" or "let me think," then gives the actual answer later. That kind of backchanneling changes what first-audio latency means.
Even with filler speech, ChatGPT seems to target around 300 ms, not 30 ms. Measuring latency to the first relevant phoneme would be more honest than counting "hmm" as a response.
GPT-Realtime-2 often starts filler at awkward moments and feels overeager, perhaps because it is bidirectional. I would rather see more latency engineering of the kind described here.
I have tuned a local Omni Voice setup for months and have never gotten below 200 ms TTFA at 24 steps. Faster output usually runs into a quality wall, so I want to test whether voice quality, cadence, and expression match the original.
We continuously compared outputs against Qwen's original implementation and did not see quality differences. Let us know how it performs in your setup.
The real win for a voice assistant is on-device, meaning cheap enough for a phone rather than an H100. Pocket TTS is very fast, and Chatterbox and Fish Audio S2 Pro are already impressive on Mac and PC, but mobile deployment is still the missing step.
We have not tested mobile, but batch size 1 and concurrency 1 should permit fast performance. Hardware-specific mobile optimizations will likely matter most.
For agents, TTS latency is only part of the delay unless the LLM emits speech tokens directly. The practical target is under 150 ms, and a 50 ms total STT plus TTS round trip would be ideal.
Qwen3-TTS supports input streaming, so the LLM can stream output into the speech model without waiting for a complete sentence. We implemented a WebSocket version under 50 ms too, though LLM time-to-first-token remains a problem.
Raw speed can make conversation feel awkward. Humans need roughly 200 ms to process audio, and an answer under 100 ms can sound like an interruption rather than a response to the final words.