In Simple Words
Doctors spend hours after every patient visit typing up notes from memory, which eats into their evenings and adds to burnout.
We built a tool that listens to the visit — only once the patient agrees — and writes a first draft of the notes automatically, clearly keeping track of what the patient said versus what the doctor said so nothing gets mixed up.
The doctor always reads, edits if needed, and signs off before anything becomes official; the system never finalizes a note on its own, so the doctor stays fully in control while saving significant time on paperwork.
At a Glance
| Metric | Value |
|---|
| Backend Workstreams | 11 (auth/RBAC → consent → recording → transcription → drafting → review/sign → addendum → export → audit → retention → backup) |
| Transcription | OpenAI Whisper, with per-segment timestamps |
| Diarization | pyannote.audio — runs locally, audio never leaves the server for speaker labeling |
| Drafting | OpenAI GPT-4o, grounded strictly in the diarized transcript |
| Backend | FastAPI, PostgreSQL/SQLAlchemy |
| Frontend | Next.js 16, TypeScript, Tailwind CSS v4, shadcn/ui (Base UI) |
| Tests | 75 backend tests passing |
| Security | AES-256-GCM at rest, Argon2id passwords, sha256 audit hash-chain, RBAC |
| Status | MVP complete, manually verified end-to-end — real patient data blocked pending an OpenAI BAA |
Architecture Highlights
Consent-Gated Recording
Whisper Transcription + Diarization
GPT-4o SOAP Drafting
Human-in-the-Loop Sign-off
Tamper-Evident Audit Trail
AES-256-GCM Encryption
Role-Based Access Control
Retention & Deletion Sweep
Independently-Keyed Backups
Full Next.js Frontend
Overview
AI Medical Scribe reduces the clinician documentation burden — the "pajama time" spent writing notes after a visit — by turning an in-visit conversation into a draft SOAP note. It is built explicitly as a copilot, not an autonomous documenter: every note that becomes part of the record was reviewed and signed by the clinician who saw the patient, and there is no code path that finalizes a note without that explicit action.
Both halves are real. The FastAPI backend implements the full clinical workflow — consent, recording, transcription, drafting, review, sign-off, post-signing corrections, export, audit, retention, and encrypted backup — and the Next.js frontend wires all of it into an actual clinician-facing UI, tested end to end against the real OpenAI Whisper/GPT-4o pipeline, real local speaker diarization, and a real Postgres database. No mocked pipeline stages, no fabricated demo data path.
The Problem
Turning a conversation into a trustworthy clinical note is not one problem — it's four, and skipping any of them produces a system that's either useless or unsafe:
Documentation Burden Gap
Clinicians lose hours a day to writing up visits after the fact — the review step needs to be faster than writing from scratch, or nobody adopts the tool no matter how good the AI is.
Trust Gap
An AI-generated note that can auto-finalize is a liability, not a copilot — the clinician has to stay the sole author of record, every time, with no exceptions baked into the code.
Consent Gap
Recording a clinical conversation without an explicit, logged consent step isn't a UI nicety — it's a compliance failure waiting to happen, and it has to be a hard, server-enforced gate, not a checkbox that can be skipped.
Speaker Attribution Gap
A flat transcript mixes up what the patient reported versus what the provider assessed — feeding that straight to an LLM risks a symptom and a diagnosis getting attributed to the wrong person in the drafted note.
The Solution
A linear, auditable pipeline — deliberately not a multi-agent graph, since a straight sequential chain with one HITL gate doesn't need that complexity:
Mic (in-app) → encrypted upload → local encrypted storage
→ queued for transcription
→ Whisper (transcription + speaker diarization)
→ GPT-4o (SOAP draft from diarized transcript, transcript-grounded)
→ clinician review UI (diarized transcript + draft side by side)
→ sign-off (HITL gate, blocking — the only path to "signed")
→ encrypted storage of the signed note
Diarization runs as part of the transcription stage, not a separate step — GPT always drafts from speaker-labelled segments, so patient-reported symptoms and provider assessments aren't mixed up by the model guessing from context.
| Layer | Technology |
|---|
| Backend | FastAPI, PostgreSQL, SQLAlchemy, Alembic |
| Transcription | OpenAI Whisper (whisper-1, verbose JSON with segment timestamps) |
| Diarization | pyannote.audio (local inference, PyAV decoding) |
| Drafting | OpenAI GPT-4o |
| Frontend | Next.js 16 (App Router, Turbopack), TypeScript, React 19 |
| UI | Tailwind CSS v4, shadcn/ui (Base UI primitives), Framer Motion |
| Data | TanStack Query, react-hook-form + zod |
| Auth | Argon2id password hashing, httpOnly/secure session cookies |
| Testing | pytest (75 tests) + Playwright-driven end-to-end verification |
How It Was Built
Built as a sequence of independently shippable workstreams, each one committed and manually tested against the real stack before the next began.
Identity, PHI Foundation & Consent Gate
Auth, RBAC (Provider / Admin), encrypted patient records, hash-chained audit logging, and the Encounter/ConsentEvent lifecycle — a consent record must exist before an encounter can leave consent_pending.
Recording, Transcription & SOAP Drafting Pipeline
Real in-app audio capture, real Whisper transcription with diarization, real GPT-4o drafting — manually verified end to end against live APIs with synthetic audio, all 14 steps of the verification plan passing.
Review, Sign-off & Post-Signing Corrections
Draft editing with revision history, an explicit non-reversible sign action, and append-only addenda for corrections made after a note is already signed — signed notes are never edited in place.
Export, Audit Surface, Retention & Backup
Signed-note export, a queryable audit trail (admin-wide and per-encounter), a retention sweep that deletes audio/transcript only after a terminal state and the retention window, and encrypted backups keyed independently from primary storage.
The Next.js Frontend
A clean clinical UI for the entire flow — login, admin patient/user management, consent, live-waveform recording, polling status stepper, side-by-side review, sign-off, addenda, and export — wired to the real backend via a same-origin proxy, no CORS layer added.
Key Engineering Decisions
No Auto-Finalize, No Exceptions
POST /encounters/{id}/note/sign is the only code path that can move a note to signed — there is no timeout, no confidence threshold, and no admin override that finalizes a note without an explicit clinician action.
Diarization Stays Local
pyannote.audio runs as a local model, not a hosted API call — the Hugging Face token only authorizes a one-time model download. One fewer third party ever touches PHI-shaped audio, by construction rather than by policy.
Same-Origin Proxy Instead of CORS
The Next.js frontend proxies API calls through its own origin rather than adding CORSMiddleware to the backend — the existing httpOnly/secure session cookie needed zero changes to stay same-origin, and one fewer cross-origin surface exists to misconfigure.
Retention Respects the Review Step, Not Just the Clock
The 30-day retention window is necessary but not sufficient — audio/transcript for an encounter still mid-review is never deleted by age alone, even past the window. Deleting source material out from under an in-progress review would undermine the one control that matters most.
Backups Keyed Independently From Live Data
Backup encryption uses its own DEK, wrapped by its own master key in the OS keystore — entirely separate from the primary storage key. A single key compromise must not expose both live data and every backup of it.
Challenges Encountered — and How They Were Resolved
A "Sign Out" button that silently did nothing.
This shadcn/ui install uses Base UI primitives, not Radix — Menu.Item takes an onClick prop, not Radix's onSelect. Passing onSelect compiled fine and the item closed the menu on click exactly like a working button, but the handler was silently never called — the logout request never left the browser. Confirmed the root cause directly against the primitive's own type definitions rather than guessing, then fixed and re-verified the actual network call fired.
A page crash that automated browser testing never caught.
DropdownMenuLabel (Base UI's Menu.GroupLabel) throws MenuGroupContext is missing if it isn't wrapped in a Menu.Group — an uncaught error that crashed the entire page the moment a real user opened the avatar menu. The scripted Playwright pass that "verified" this flow had clicked the trigger button but never actually asserted the dropdown's contents, so it missed the crash entirely. Caught only when the actual end user clicked through the app themselves — a reminder that a script proving a button is clickable is not the same as proving the feature works.
A leftover scaffold file silently winning the home route.
create-next-app's default src/app/page.tsx was never deleted after the real src/app/(app)/page.tsx was added inside a route group — both resolved to /, and the placeholder template silently won, rendering "To get started, edit page.tsx" instead of the login-gated app. Caught by inspecting the raw server-rendered HTML, not by assumption.
Base UI's Select passes a nullable value, Radix's doesn't.
onValueChange on this install's Select is typed (value: string | null) => void, not a guaranteed string — code written against the Radix-shaped assumption failed to compile, and a form left without a matching defaultValues entry threw a benign-looking but real "uncontrolled → controlled" console warning the first time a value was picked. Fixed by checking the actual generated component source under @base-ui/react/* before wiring new interactions, rather than assuming API parity with a different library the shadcn CLI has historically shipped.
Security & Compliance Posture
What's actually built today, stated as current-state, not aspiration:
Consent Logged Before Recording Ever Starts
AES-256-GCM Encryption at Every Hop
Argon2id Passwords, HttpOnly/Secure Sessions
sha256 Hash-Chained, Tamper-Evident Audit Trail
Human-in-the-Loop — No Auto-Finalize Path Exists
RBAC Enforced Server-Side on Every Endpoint
Independently-Keyed Encrypted Backups
Stated plainly: real patient audio and transcripts are blocked from reaching the Whisper/GPT APIs until a signed Business Associate Agreement and "Modified Retention" account provisioning are confirmed with OpenAI — synthetic and de-identified data only until then. SSO/MFA is not built yet (documented deviation from a Zero-Trust baseline). No professional security audit has been performed.
Engineering Validation
75
Automated backend tests passing (pytest)
Live Pipeline
Real OpenAI Whisper + GPT-4o + local pyannote diarization, real Postgres — no mocked stages
RBAC Live-Verified
Admin blocked from every encounter route — returns not-found rather than leaking existence
Full Manual Pass
Login → consent → record → transcribe → draft → review → edit → sign → addendum → export, driven end-to-end through the real browser by the actual end user
A signed note generated by the real pipeline from a spoken clinical conversation — every stage of the status stepper reflects an actual completed backend step, not a simulated one.
The provider's encounter list, dark mode — theme support was built in from the start, not bolted on.
Honest Limitations
Stated deliberately, because a system that hides its limitations is worth less than one that states them:
Not yet used with real patient data — blocked on purpose until an OpenAI Business Associate Agreement and Modified Retention provisioning are formally in place.
No SSO/MFA yet — username/password login only, a documented deviation from the Zero-Trust baseline.
No automated frontend test suite yet — every frontend workflow was manually verified end-to-end against the real backend instead of by a scripted regression suite.
Audit-log deletion after the 7-year retention target is flagged, not built — deleting the oldest hash-chained entries would break tamper-evidence for everything after them without a re-anchoring design first.
Single-server, local-first deployment only — no cloud or multi-region hosting built yet, by original scope decision.
No professional third-party security audit has been performed.
What This Case Study Demonstrates
- Full-stack ownership — a real FastAPI backend and a real Next.js frontend, wired end-to-end against each other, not a backend exercised only through Swagger.
- Human-in-the-loop enforced by construction, in a domain where "auto-finalize" is exactly the wrong shortcut to take.
- A live-testing discipline that catches real bugs — a silently-dead logout handler, a page-crashing menu, a shadowed route — that unit tests, and even scripted browser automation, missed until an actual human clicked through the app.
- Security and compliance posture reasoned about explicitly (consent gate, encryption at every hop, tamper-evident audit, independently-keyed backups) and labeled honestly as current-state versus not-yet-built, throughout.
- Comfort adopting an unfamiliar library's real API (Base UI, not the more common Radix) by reading its actual type definitions rather than assuming parity with a more famous alternative.
Building a Healthcare AI System of Your Own?
Whether it's clinical documentation, an EMR copilot, or any workflow that needs a human firmly in the loop, we're always interested in discussing secure, HIPAA-aligned AI architectures.
