Most voice bots feel like a walkie-talkie: talk, wait, reply, wait. Vāṇī (Sanskrit for “voice”) is built to feel like a conversation: you can talk over it and it stops and listens, and it starts following you before you finish a sentence. The same engine runs in the cloud or fully offline.
The bet: every part is a socket
Most teams pick one speech-to-text vendor, one language model and one voice, and build the product around that choice. Change your mind later and it is a rewrite. Vāṇī treats each step as a socket. Any vendor’s tool that fits plugs in, and the part running the conversation never needs to know which. A vendor registers with one line, and a settings file picks who fills each socket:
@register("tts", "kokoro")
class KokoroTTS(TTSBase): ...
stt: { provider: deepgram }
llm: { provider: anthropic, model: claude-3-5-sonnet }
tts: { provider: kokoro, voice: af_heart }Adding a vendor is one new file. That one idea gives three commercial promises from one codebase: the fastest reply in the cloud, fully offline with no API keys and nothing leaving the network, or the client’s own OpenAI or Azure contract.
| Socket | Cloud | Local, offline |
|---|---|---|
| Speech-to-text | Deepgram | Whisper, Moonshine |
| Language model | OpenAI, Anthropic, Azure OpenAI | Ollama |
| Voice | Deepgram Aura | Kokoro |
| Audio transport | Browser, WebRTC through LiveKit (real echo cancellation) | Local mic and speaker |
Where the difficulty lives, and four ways to listen
A conversation is a relay race with seven handoffs: mic, voice detection, turn detection, speech-to-text, language model, voice and speaker. None of the individual steps is hard, because calling a model is a solved problem. The difficulty is the coordination between them: queues, sentence boundaries, and knowing when someone is done talking and not just pausing to think.
1 · Record, then send: A voice note. Understanding starts only once you stop, and the engine waits about 700 ms of quiet to be sure you are done.
time →
2 · Record and stream: It starts understanding while you are still talking, so almost nothing is left to work out when you stop. It still waits about 700 ms of quiet.
time →
3 · Plus turn detection: It learns to tell “I’m thinking” from “I’m done” instead of guessing from silence, so the fixed wait disappears.
time →
4 · Plus interruption: A real conversation. Talk over it and it stops and listens, instead of finishing its thought.
time →
Schematic, not measured: bar lengths are illustrative. The team has not published latency figures; the evaluation pipeline below is how they are meant to be measured.
What the team learned
- Echo looks exactly like an interruption. With noise-cancelling hardware, interrupting the bot is easy. Without it, the bot’s own voice bleeding into its own microphone looks just like a person talking over it. Three layered checks tell the two apart: higher confidence, a warm-up window, and requiring sustained speech. WebRTC with real echo cancellation is the stronger fix.
- “Async everything” does not save you from the operating system. A real concurrency bug had the microphone thread starving the rest of the system, so the bot went silent. It was fixed, and it is a reminder that async code still meets OS-level threading conflicts.
- Speak sentences, not fragments. The model writes a reply a word fragment at a time, and speaking fragments as they arrived sounded robotic. The engine holds text until it forms a full sentence.
- Overlap listening and speaking. They once ran strictly one after the other, which felt sluggish. Now transcription starts while you are still talking, and the next sentence is prepared while the current one plays.
The barge-in gate, in the order a candidate interruption has to pass:
- Speech heard
- Higher confidence
- Past the warm-up window
- Sustained, not a blip
- Stop playback
Proving a swap did not make it worse
If any piece can be swapped, “sounded fine when I tried it” is not enough. The evaluation pipeline, in progress, scores each piece against a known-good baseline, so a developer who touches one piece gets a number, not a guess. That is what turns “you can switch vendors anytime” into “and we can prove it did not get worse”.
| Module | Compared against baseline on |
|---|---|
| Speech-to-text | Word error rate and latency |
| Language model | Response latency; response-quality scoring planned |
| Voice | Latency; naturalness scoring planned |
| Voice detection | Speech and silence detection accuracy |
| Turn detection | False-cutoff and false-wait rates |
| Interruption | Response latency and false-positive rate |
Where it stands
The engine is built and working end to end. What is missing is what turns an engine into a product.
| Layer | Status |
|---|---|
| Pluggable pipeline and provider registry | Done |
| Streaming end to end | Done |
| Interruption handling | Done |
| WebRTC transport (echo cancellation, mobile) | Done |
| Evaluation pipeline | In progress |
| Embeddable widget (one script tag) | Not started |
| Multi-tenant hosting, auth and metering | Not started |
The goal: one script tag gives anyone a production-quality voice agent, with the persona and providers of their choice and the hosting handled for them.