Speech, Scored in Real Time

Clinicians rehearse procedure endlessly and conversation almost never. A training tool has to hold a real spoken exchange, then explain — credibly — what went well and what did not.

Latency is the whole product. A voice assistant that pauses to think stops being a conversation.

I built the full React 19 and TypeScript frontend for a real-time multimodal application — speech-to-speech plus text — on Firebase, with TanStack Query for server state and Zustand for session state.

The streaming UI pairs context-aware voice selection with an LRU audio cache, cutting perceived latency where users feel it most: the gap before the first syllable.

For text mode I integrated AI Elements, the Next.js AI chat framework, rather than rebuilding conversational primitives from scratch.

The front-end evaluation console visualises automated LLM-as-judge scoring across six weighted rubric dimensions, each score carrying the evidence quotes that produced it — so a trainee can trace judgement back to what they actually said.

PostHog analytics and error reporting are wired through the app, closing the loop between session behaviour and product decisions.