The problem
High-stakes conversational skills — de-escalating an angry caller, giving direction to someone who can't think straight, listening through ambiguity — are almost impossible to practice safely. Role-play with a human coach is expensive, inconsistent, and rarely available on demand.
Doing this with voice AI is hard: the agent has to hold a consistent persona, respond in real time with natural turn-taking and barge-in, and stay in character under pressure while the human drives the call.
The approach
We built a production web app where the player is a 911 dispatcher (or, in one scenario, an undercover officer) on a live voice call with an LLM-driven character. Each scenario targets a different conversational skill axis — empathy under ambiguity, direction-giving, listening through anger, bluffing under asymmetric information.
Each scenario runs a single-leg real-time voice loop (speech-to-text, LLM reasoning, text-to-speech) with a per-scenario persona and system prompt. One scenario is fully procedurally generated: a fresh case file (names, crime type, code phrase, deal terms) is produced on every session so it never plays the same way twice.
After each call, a supervisor-style debrief is generated from the transcript, grounded in real domain craft (behavioral-change negotiation frameworks, fire-service SOPs, repeat-caller dispatch practice, undercover comms).
What I built
- Four playable voice scenarios selectable from a lobby, each with its own AI persona, in-call visual surface (surveillance grid, floor plan, records, notes), and scenario-specific decision action
- Real-time voice loop per scenario via ElevenLabs Conversational AI, with live status (listening / speaking / thinking) and natural barge-in
- A procedurally generated scenario where an LLM authors a unique case file each session, plus an 'evidence extracted' scorecard tracking which critical facts the player got the target to confirm
- LLM-generated post-call debriefs grounded in real crisis-communication and dispatch best practices
- One-shot generation of scenario visual assets (CCTV stills, floor plans) via an image model pipeline
- Per-call state machine (timeline beats, action chips, auto-end), post-call feedback capture, and cohort tagging for playtests
- SvelteKit
- Svelte 5
- ElevenLabs Conversational AI
- Anthropic Claude (Sonnet + Haiku)
- Google Gemini Imagen 4
- TypeScript
- Railway
Result
A live, publicly playable real-time voice AI product deployed to production with four distinct scenarios.
- Runs in production on a custom domain with real-time browser voice
- Procedurally generated scenario proves parametric, per-session persona generation — not scripted playback
- Debriefs are grounded in cited domain frameworks rather than generic feedback