In Simple Words
Doctors lose hours every day retyping the same patient details into multiple screens, and because the information is spread across so many separate tabs, it takes real mental effort just to piece together the full picture of what's going on with a patient.
We built a system that automatically reads a patient's full medical history, organizes it around each specific health issue instead of scattering it by data type, and pre-fills a draft form for the doctor — double-checking every fact against the real record and hiding any private patient details before anyone outside the care team can see them.
This has not yet been rolled out in a real hospital, so exact time savings have not been measured, but it is built to take over the manual data-entry work that currently costs a doctor 2 to 4 hours a day, and it has already passed hundreds of automated checks confirming it works reliably and safely.
At a Glance
| Metric | Value |
|---|
| Agents | 11 (LangGraph StateGraph, RBAC-gated) |
| Standards | HL7 v2, FHIR R4 |
| Backend | FastAPI, PostgreSQL/SQLAlchemy |
| AI | OpenAI GPT-4o |
| Vector DB | Qdrant (RAG) |
| Tests | 426 passing, offline by default |
| Security | JWT (HS256), AES-256-GCM, sha256 audit hash-chain, RBAC |
| Status | Reference implementation — not yet deployed to a real hospital |
Architecture Highlights
11-Agent LangGraph Pipeline
React Clinician UI
FHIR R4 Native
HL7 v2 Ingestion
RAG (Qdrant)
Human-in-the-Loop
Tamper-Evident Audit Trail
Verified JWT Auth
AES-256-GCM Encryption
Role-Based Access Control
Overview
EMR Agentic AI is a reference implementation of the technology stack behind AI-assisted clinical documentation: a multi-agent pipeline that turns a doctor's raw, fragmented FHIR chart into a single, condition-centric pre-fill form — de-identified, evidence-grounded, hallucination-checked, and RBAC-gated, before a clinician ever sees it. It was built to demonstrate the exact skill set a healthcare AI engineering role requires: clinical workflow translation, FHIR/HL7 interoperability, agentic pipeline architecture, and the judgment to build a system that is trustworthy, not just impressive, in a domain with near-zero tolerance for a hallucinated diagnosis or a leaked identifier.
Every component is real and runs end to end against a local Postgres database and either bundled synthetic FHIR data or a live HAPI FHIR Docker server — no mocked responses over fake data.
The Problem
A doctor loses 2–4 hours a day to manual EMR data entry, and the reasons don't fix themselves independently:
UI Gap
The same data is re-typed across 6+ screens because nothing pre-populates from what the system already knows.
Processing Gap
Feeding a raw chart straight into an LLM produces hallucinations; data has to be cleaned and structured first.
Intelligence Gap
An LLM answer is only useful if it's traceable back to a real source in the chart.
Cognitive Load Gap
Data organized by type (Labs tab, Meds tab) forces the clinician to mentally reassemble the picture; real clinical reasoning is organized by condition.
The hard part is that closing these gaps requires an AI pipeline touching real patient data, in a domain with near-zero tolerance for the two things generative AI is worst at by default: making things up, and seeing information it shouldn't have touched.
The Solution
An 11-agent LangGraph StateGraph, RBAC-gated so each role only runs the agents its permissions cover:
| Layer | Technology |
|---|
| Backend | FastAPI, PostgreSQL, SQLAlchemy |
| Orchestration | LangGraph (StateGraph) |
| AI / Reasoning | OpenAI GPT-4o |
| Retrieval | Qdrant (RAG) |
| Interoperability | HAPI FHIR (R4), hl7apy (HL7 v2) |
| Auth | JWT (python-jose, HS256) |
| Testing | pytest (426 tests) |
How It Was Built
The build was sequenced across five phases, each clearing a named architectural gate before the next workstream was allowed to build on it.
Phase 1 — Foundation
A pluggable FHIR client, a Phase-1 stub pipeline, a condition-centric response schema shaped by workflow discussions with practicing clinicians (including Dr. Sameer and another practicing doctor) — clinician-informed design input, not a formal clinical validation study — and a real React/TypeScript clinician UI wired to the API.
Phase 2 — PHI De-identification + Audit Trail
Real 18-identifier stripping and free-text NER redaction, a HIPAA audit log, and stable pseudonymous patient identity via an HMAC blind-index lookup.
Phase 3 — Orchestration + RAG + Risk Detection
Migrated to a LangGraph StateGraph with RBAC-driven routing; added Timeline Chronicler, Qdrant-backed Evidence Retrieval, a Clinical Risk Detector, and a real HAPI FHIR Docker backend.
Phase 4 — Reasoning + Verification + Live Terminology
Added evidence-grounded differential diagnosis and a Verification Agent with a numeric confidence score; swapped hardcoded ICD-10/LOINC dictionaries for a live terminology service — which surfaced a real coding-safety bug (see Challenges below).
Phase 5 — Verified Auth + HL7 + Security Hardening
Replaced an honor-system role parameter with a verified JWT; built a from-scratch HL7 v2 → FHIR translator; closed with an RBAC allow-list fix, a tamper-evident audit hash-chain, rate limiting, and security headers.
Key Engineering Decisions
PHI Boundary ≠ UI Boundary
PHI is always stripped before anything reaches the LLM, regardless of caller role. Whether a clinician's own UI may show a patient's real name is a separate, narrowly-scoped RBAC decision — conflating the two is a common healthcare-AI design mistake.
Stable Identity Without a Queryable MRN
The same patient needed the same pseudonymous ID across visits, without ever storing a queryable plaintext MRN. Solved with an HMAC-SHA256 blind index — an equality lookup and uniqueness constraint without ever decrypting the encrypted MRN column.
RBAC as an Allow-List, Not a Deny-List
A deny-list only rejected one role, silently letting a later-added machine-credential role fall through to the full patient list. Fixed by inverting to an explicit allow-list, with every denial audited — a deny-list fails safe until the next role is added; an allow-list fails safe by construction.
Audit Trail Attributable to a Person, Not a Role
Roughly 30 call sites across the pipeline's audit-writing code were passing the role string ("doctor") as the acting user's ID — the audit trail could show what happened and what role did it, but never who. In a real HIPAA audit trail, that's the difference between "a doctor edited this record" and "Dr. Jones edited this record." Fixed by threading a real user_id, distinct from role, through the entire pipeline.
Verification as a Separate Stage, Not a Prompt Instruction
Rather than asking the Reasoning Agent to "please cite your sources," a separate Verification Agent independently cross-checks every claim against the actual retrieved evidence — deterministically first, falling back to an LLM faithfulness judgment only when the deterministic check can't resolve it. A model grading its own homework is a weaker safety property than a second, narrower system whose only job is checking the first one's work.
Challenges Encountered — and How They Were Resolved
A ResponseValidationError invisible to the entire test suite.
GET /prefill/{id} degraded correctly to a stub response for a low-access role, but FastAPI's response_model validated independently of what the handler returned and rejected it with a 500 — invisible to every test because they all called the pipeline function directly, bypassing FastAPI's response layer. Caught only by adding real HTTP tests. The lesson: verify through the interface a user actually touches, not just the function underneath it.
A same-day tamper-evidence false positive, caught during this project's own live-verification pass.
After adding a sha256 hash chain to the audit log, the first live run against real Postgres flagged every row as tampered. Root cause: the hash was computed from a timezone-aware timestamp at insert time, but the column itself is timezone-naive — Postgres silently drops the timezone, so re-reading the row produces a different string than what was hashed. Fixed by normalizing the timestamp consistently before hashing, with a regression test pinning the exact case. The lesson: a live pass against a real database is not optional even when unit tests are green.
A live terminology-API ranking bug that would have silently miscoded new conditions.
After swapping a hardcoded ICD-10 dictionary for a live NLM Clinical Tables lookup, live testing surfaced spurious "ICD-10 conflict" flags against already-correctly-coded conditions. The API's raw #1 search hit doesn't reliably rank the generic/canonical code first — the same bug would have silently miscoded any new condition with nothing existing to catch the mismatch against. Fixed with a safety-conservative disambiguation pass that penalizes subtype/etiology qualifiers, and raises an explicit "ambiguous — needs manual review" flag rather than guessing when it still can't settle on one candidate.
A field silently reset by a framework default, several agents downstream.
Pydantic copies list fields at model construction rather than sharing a reference, so data_quality_flags — captured once by the Pre-fill Agent — silently stopped reflecting any flag a later agent appended afterward. Live-caught: a redacted query's "PHI scrubbed" flag never appeared in the API response, even though the redaction itself worked correctly — only the audit trail of it silently didn't. Fixed by re-stitching the flags list explicitly at the end of every downstream agent that might append to it.
Security & Compliance Posture
Mapped explicitly against HIPAA's technical safeguards:
PHI De-identification Before Any LLM Call
RBAC on Every Endpoint, from a Verified JWT
sha256 Hash-Chained Audit Trail
AES-256-GCM Encrypted PHI Mapping Table
Human-in-the-Loop by Construction
Security Headers + Per-IP Rate Limiting
Just as important, this is stated plainly: TLS termination, KMS/HSM-managed keys, storage-layer encryption at rest, and MFA/SSO are deliberately deferred, built only when a real deployment target exists rather than speculatively. There has been no professional security audit yet.
Engineering Validation
426
Automated tests passing, zero network/Docker/API-key dependency by default
Live-Docker
RBAC, audit attribution, hash chain, rate limiting, and headers verified against real Postgres + HAPI FHIR
4 Gates
Named architectural gates cleared with live evidence before each dependent workstream proceeded
Manual UI Verification
Every implemented v1.0 workflow — login, pre-fill, RBAC, Reasoning/Verification output, Risk Detector alerts — manually driven end-to-end through the browser against the real backend
Regulator-facing audit endpoint with role-based access control and patient-level filtering.
Tamper-evident audit chain verification confirming log integrity across the complete audit trail.
Honest Limitations
Stated deliberately, because a system that hides its limitations is worth less than one that states them:
Not yet clinically validated at scale by practicing clinicians, and not yet deployed to a real hospital — broader clinician validation is the next planned milestone.
Confidence/accuracy targets (≥99.5% faithfulness, ≥85% pre-fill accuracy) are architectural targets, not yet measured against real clinical use.
Database encryption at rest is local/dev-only — explicitly not production-ready.
A known, accepted concurrent-insert race in the audit hash chain — acceptable for single-clinician demo scope, would need addressing before real concurrent multi-clinician load.
No federated SSO/OIDC yet (local username/password login only).
The React/TypeScript clinician UI is fully implemented and feature-complete through Phase 5/v1.0 (login, condition-centric pre-fill, HITL review, Risk Detector alerts, Reasoning/Verification output) — every workflow manually verified end-to-end against the real backend. What's missing is an automated frontend test suite, not frontend functionality; a manual checklist stands in for one.
What This Case Study Demonstrates
- Clinical workflow understanding translated directly into architecture — condition-centric design informed by real clinician workflow discussions, refined through later development, not guessed.
- FHIR R4 and HL7 v2 interoperability built from the standards themselves.
- An agentic pipeline with real role-based routing, not a single mega-prompt.
- A security posture mapped explicitly to HIPAA, stated honestly.
- The discipline to catch real bugs through live testing — not just green unit tests — and document them rather than quietly patch and move on.
Building a Healthcare AI System of Your Own?
Whether you're building an AI-assisted EMR, a clinical workflow automation platform, or a healthcare interoperability solution, we're always interested in discussing secure, agentic AI architectures.
