AssemblyAI Voice Agent Hackathon
MediScribe Live
A voice-first ER assistant built on AssemblyAI: bedside voice triage with live tool calls, and an ambient scribe that writes SOAP notes where every line links back to what was said.
- AssemblyAI APIs
- 3
- Voice modes
- 2
- Export
- FHIR R4
The problem
A note you can't check is a note you can't trust
Emergency clinicians lose a large part of every shift to typing EHR notes. AI scribes can help, but a clinician cannot trust a note when they cannot see where each line came from. Invented details, like a default 'no known allergies', are dangerous.
The build
Two voice modes, one chart, every line sourced
Three AssemblyAI products do three separate jobs. The Voice Agent API runs bedside triage with barge-in and JSON-schema tool calls (record_patient_intake, check_drug_interaction, flag_critical_vital). Realtime STT (Universal-3.5 Pro, domain medical-v1, speaker labels) transcribes the consultation. The LLM Gateway drafts the SOAP note. The server numbers every transcript turn and checks each citation against the real turn text. Wrong citations are fixed or dropped, and items without a source are flagged 'NO SOURCE - VERIFY'.
The note stays a DRAFT until a clinician enters their name, acknowledges any unsourced items, and signs. Signing turns the FHIR Composition from preliminary to final with an attester. The voice panel shows real response latency, measured in the browser from end of speech to the first agent audio.
Bedside Voice Triage
Full-duplex voice agent with barge-in. Calls tools mid-conversation and updates the chart, ESI score and drug alerts live.
Ambient Scribe
Universal-3.5 Pro with the medical-v1 domain. Separates Doctor and Patient and catches drug names and clinical terms.
Source-Linked SOAP
Every note item cites transcript turns, checked on the server. Clinician sign-off, then HL7 FHIR R4 export.
Results
What shipped
- Submitted to the AssemblyAI Voice Agent Hackathon (lablab.ai, September 2026). Built solo as team Neuron Burners.
- Uses three AssemblyAI products in one app: Voice Agent API, Realtime STT and LLM Gateway.
- Nothing is invented: topics not discussed say so, allergies never default to NKDA, and a diagnosis appears only when the clinician states it.
- Exports an HL7 FHIR R4 Bundle. The note moves from preliminary to final only after a named clinician signs.
- The API key stays on the server. The browser gets short-lived tokens and connects straight to AssemblyAI over WebSockets.
A hackathon prototype, not a medical device. The drug-interaction checker uses a small demo rule set.