In Simple Words
When someone asks a general AI chatbot a health question, it often answers confidently even when it's really just guessing, and there's no way to know whether that answer is backed by real medical information or made up on the spot.
We built a chatbot that only answers using a trusted, curated set of medical documents, clearly shows which source each answer came from, tells the user how confident it is in that answer, and honestly says it doesn't know rather than inventing something.
The result is a far more trustworthy system — every answer can be checked against its source, and the chatbot refuses to guess when the evidence isn't strong enough, instead of giving a confident-sounding wrong answer.
At a Glance
| Metric | Value |
|---|
| Pipeline | Retrieve → Re-rank → Confidence-Tier, kept as three separate stages |
| Backend | FastAPI (/chat endpoint — the real deliverable/API) |
| Demo UI | Streamlit, calling the FastAPI backend |
| Vector DB | ChromaDB, behind a swappable VectorStore interface + factory |
| AI | OpenAI (embeddings + generation) |
| Corpus | 129 chunks — 35 PubMed abstract chunks + 94 curated Kaggle Q&A chunks |
| Tests | 56 passing, zero network calls in the default suite |
| Status | All 9 build phases complete, thresholds calibrated against the real corpus |
Architecture Highlights
Retrieve / Re-rank / Tier Separation
4-Tier Confidence Model
Conflicting-Evidence Heuristic
Provenance-Aware Citations
Calibrated Thresholds
Deterministic Disclaimer
Swappable Vector Store
Overview
Medical RAG Chatbot is a retrieval-augmented generation system that answers health questions strictly from a curated, synthetic/public medical corpus — PubMed abstracts and a manually-vetted subset of a Kaggle medical Q&A dataset. Every response carries citations back to the retrieved source material, an explicit evidence-quality tier, and a heuristic flag when the top sources disagree. It was built as the first project in a healthcare AI portfolio pipeline, chosen for the fastest build time and the highest Upwork demand, to demonstrate the exact judgment a healthcare RAG engagement requires: trust-first design, honest confidence signaling, and the discipline to refuse an answer rather than fabricate one.
The core RAG pipeline, the FastAPI /chat endpoint, the Streamlit demo, and the corpus ingestion/calibration scripts are all real and run end to end against the actual ingested corpus — no mocked retrieval, no canned responses.
The Problem
Generic LLM chatbots answer medical questions from parametric memory, and that creates a real liability even in low-stakes patient-education use cases:
No Citations
A confident-sounding answer with no traceable source — no way for the user to check where it came from.
No Trust Signal
No way to tell a well-supported answer from a confident-sounding guess — every response reads with the same authority.
No Honest "I Don't Know"
No distinction between the model genuinely not knowing and the model quietly making something up to fill the gap.
This project builds a RAG pipeline where every claim is traceable to a retrieved source, and every response is honest about how much to trust it.
The Solution
Retrieval, re-ranking, and confidence tiering are kept as three separate, testable stages rather than one conflated relevance score — a common RAG mistake this system deliberately avoids:
| Stage | Job |
|---|
| Retrieve | Embed the query, search ChromaDB, return candidate chunks |
| Re-rank | Weight similarity + evidence level + recency to decide which chunks to trust |
| Tier | Decide how much the final answer deserves — VERIFIED / LOW_CONFIDENCE / INSUFFICIENT_EVIDENCE / NO_MATCH |
Four confidence tiers, each shown to the user with the answer:
VERIFIED
Top retrieved chunk carries HIGH/MEDIUM evidence — "verified" means backed by higher-quality evidence, not just high similarity.
LOW_CONFIDENCE
Multiple independent sources agree, but all are lower-evidence — an answer is given, with the evidence level shown honestly.
INSUFFICIENT_EVIDENCE
The corpus has something related, but not enough to answer confidently — a deliberately distinct failure mode from NO_MATCH.
NO_MATCH
The corpus has nothing relevant — the system says so and refuses to fabricate an answer outside its knowledge base.
A heuristic (non-LLM) CONFLICTING_EVIDENCE flag can accompany any tier when top sources span multiple provenance types with a wide score spread — surfaced to the user rather than auto-resolved, since auto-resolution would need its own LLM call and cross into agentic behavior this v1 deliberately defers.
| Layer | Technology |
|---|
| Backend / API | FastAPI |
| Vector Store | ChromaDB, behind a VectorStore interface + factory (in-memory for tests) |
| AI | OpenAI (embeddings + generation) |
| Demo UI | Streamlit |
| Testing | pytest (56 tests, zero network calls by default) |
Key Engineering Decisions
Provenance-Aware Citations, Not Bare Links
Every citation carries source type, evidence level, and publication year — because a citation should say where an answer came from, and the evidence metadata should say how much to trust it.
Thresholds Calibrated Against the Real Corpus
Confidence-tier boundaries (tier_no_match_floor, tier_verified_floor) were measured from the actual embedding-score distribution of the ingested corpus via scripts/calibrate_thresholds.py, not hardcoded tutorial defaults.
Hallucination Guard + Deterministic Disclaimer
The system prompt explicitly instructs the model to answer only from retrieved evidence and say so when evidence is missing; the medical disclaimer is appended in code after generation on every response — not left to the LLM to remember, and not confined to the UI layer.
Swappable Vector Store From Day One
Retrieval sits behind a VectorStore interface + factory (in-memory for tests, ChromaDB for the running system) — moving to a managed vector DB for a client with different scale needs is a config change, not a rewrite.
No LangChain, No Agents SDK — Until a Requirement Justifies It
A single-turn RAG pipeline (embed query → search → prompt) doesn't need an orchestration framework. LangChain or the OpenAI Agents SDK would earn their place for a triage agent, a self-critique/retry loop, or multi-agent routing — none of which v1 needs. Judgment about when not to add complexity, not just what to add.
An Honest Engineering Call: What's Not Built, and Why
PubMed under-retrieval on patient-phrased queries.
The corpus's Kaggle Q&A content is phrased like patient questions and out-embeds PubMed's dense scientific prose on shared topics (diabetes, hypertension, asthma) — at the current corpus scale (94 Kaggle chunks vs. 35 PubMed chunks), PubMed content can be crowded out of the retrieved candidate set before re-ranking ever sees it, regardless of retrieval_top_k. Bumping retrieval_top_k to 4x the shipped value was tested first as the cheapest fix and empirically disproven — a bigger pool just returns more Kaggle.
The short-term fix that's live: hardening tiering, not patching retrieval.
Rather than ship a silent inaccuracy, the tiering logic (_has_sufficient_coverage) was hardened so an all-LOW-evidence result only reaches LOW_CONFIDENCE — never a false INSUFFICIENT_EVIDENCE — when enough independent sources genuinely agree within a tight score spread. A companion guard (_top_evidence_qualifies_for_verified) stops a high-scoring all-Kaggle result from reaching VERIFIED on similarity alone, so "verified" keeps meaning backed by higher-quality evidence.
The real fix — per-source retrieval quotas — is scoped and deliberately deferred.
The recommended long-term direction is querying the vector store separately per source type and merging before re-ranking, so HIGH-evidence content always reaches the evidence-weighted rank stage instead of being crowded out at the raw-ANN stage. It's deferred because the tiering-side fix already prevents the two failure modes that matter at portfolio scale (false INSUFFICIENT_EVIDENCE, false VERIFIED), and per-source quotas add real complexity — query fan-out, per-source tuning — that isn't worth carrying until the corpus grows or a real client requirement calls for it.
This is the kind of trade-off documentation a client should expect on a real engagement — not just a working demo, but a record of what's solid, what's a known limitation, and why, kept in the project's decision log rather than silently worked around.
Data & Compliance Posture
Stated plainly, because a system that hides its posture is worth less than one that states it:
Synthetic / Public Data Only — No Real Patient Data
PubMed Abstracts via NCBI E-Utilities
Manually-Vetted Kaggle Q&A Subset (CC BY 4.0)
Medical Disclaimer on Every Response
MIMIC-III Demo was evaluated for inclusion and dropped from scope after discovering its public demo release ships an empty NOTEEVENTS table — no free-text clinical notes, only structured tables. Real clinical notes require full credentialed MIMIC-III access (CITI training + a signed Data Use Agreement), which is out of scope for a portfolio build; this is documented in the project's decision log rather than silently worked around. Real patient data of any kind would require encryption, access control, and a signed BAA — not applicable to this project, and not claimed.
All Four Confidence Tiers, in the Live Demo
Every query below was run against the live API and confirmed to produce the tier shown:
VERIFIED + CONFLICTING_EVIDENCE — "Can asthma inhalers cause long-term dependency?" A HIGH-evidence PubMed citation tops the evidence cards alongside lower-scored Kaggle cards.
LOW_CONFIDENCE — "What is a normal blood pressure reading for an adult?" Answer given (120/80 mmHg), citations all lower evidence level, stated honestly.
INSUFFICIENT_EVIDENCE — "What are the early warning signs of Parkinson's disease?" Only 3 weak/scattered citations; the answer says evidence is insufficient to fully answer.
NO_MATCH — "What is the treatment for a broken toe?" The system states it could not find relevant information rather than guessing — empty citations list.
Engineering Validation
56
Automated tests passing, zero network calls in the default suite
129
Chunks ingested — 35 PubMed abstract chunks + 94 curated Kaggle Q&A chunks
Calibrated
Confidence thresholds measured against the real corpus's embedding-score distribution
9 Phases
Full build plan complete, backend + Streamlit demo both run end to end against the live corpus
Honest Limitations
Stated deliberately, because a system that hides its limitations is worth less than one that states them:
On-topic patient-phrased queries can under-retrieve PubMed chunks at the current corpus scale — mitigated on the tiering side, the underlying retrieval fix (per-source quotas) is deliberately deferred. See "An Honest Engineering Call" above for full detail.
MIMIC-III Demo was dropped from scope — its public demo release ships no free-text clinical notes, only structured tables.
This is an educational/portfolio project — synthetic and public data only, not for clinical use, not medical advice.
Not connected to any real EHR, clinician workflow, or patient-facing deployment — a reference implementation of the RAG trust pattern, not a shipped product.
No professional security or clinical-accuracy audit has been performed.
What This Case Study Demonstrates
- RAG systems designed around trust, not just relevance — critical for any regulated or high-stakes domain (healthcare, legal, financial).
- Judgment about when not to add complexity (no LangChain, no agent framework, no managed vector DB) until a concrete requirement justifies it — and a clear plan for when that judgment would change.
- Comfort working with provenance/compliance-adjacent constraints (synthetic data only, no PHI, no overclaimed compliance posture) even outside a regulated client engagement.
- The discipline to document a real limitation and its deferred fix in detail rather than quietly patching around it or shipping a silent inaccuracy.
Building a Healthcare AI System of Your Own?
Whether you're building a patient-education chatbot, a clinical knowledge-retrieval tool, or any RAG system where trust matters as much as relevance, we're always interested in discussing secure, evidence-grounded AI architectures.
