Philterd

Talk to an Expert

Tell us about your stack and the privacy problems you're trying to solve. We typically respond within one business day.

Prefer to skip the form? Pick a time on our calendar →
or send a message

← All posts

Building a HIPAA-Compliant Medical Chatbot

By AI · LLMs · Philter

Every health system in 2026 is building or evaluating an internal medical chatbot — a Q&A interface over chart notes, drug references, clinical guidelines, or operational documentation. The reasoning is straightforward: physicians and care teams spend hours a day searching for information that should be one question away. The technology is straightforward too: it's a RAG system.

The HIPAA story, however, is decidedly not straightforward. A generic RAG chatbot built with off-the-shelf components fails HIPAA at multiple points, and the failures aren't subtle. This post is the architectural blueprint for the version that passes — a medical chatbot that satisfies Safe Harbor de-identification at ingestion, defends against PHI leakage at retrieval and inference, and produces the audit artifacts your compliance team will be asked to show.

Why a generic RAG chatbot fails HIPAA

Walk through the standard tutorial-grade RAG pipeline and count the PHI exposure points:

  1. Ingestion. Source chart notes get chunked and embedded. The raw text is stored in the vector database alongside the embeddings as the "metadata" or "documents" payload. Anyone with read access to the vector store has read access to PHI.
  2. Embedding model. If you're using a hosted embedding API (OpenAI, Cohere, Google), the chunk text is sent to the model provider to compute embeddings. PHI leaves your perimeter on every ingestion.
  3. Vector store hosting. Hosted vector databases (Pinecone, Weaviate Cloud) put your embeddings on someone else's infrastructure. Embeddings are partially invertible; this is its own PHI exposure path.
  4. Retrieval. Top-K retrieved chunks get pasted into the LLM prompt. The retrieved content is raw chart text.
  5. Inference. The prompt goes to the LLM provider. PHI lands in their logs, retained per their policy.
  6. Response. The model can hallucinate PHI (especially when asked to summarize), which then gets returned to the user without scanning.
  7. Conversation history. Multi-turn chats append history to each prompt, multiplying the PHI exposure with every turn.

Any one of these is a HIPAA breach in waiting. The fix is to address each one explicitly — not with a single sweeping change, but with a layered set of decisions about where the data flows.

The compliant architecture

  EHR / chart notes ──▶ Philter (Safe Harbor) ──▶ Self-hosted
                       (ingestion-time redaction)   embedder
                                                       │
                                                       ▼
                                          Self-hosted vector store
                                          (BAA-compliant infrastructure)
                                                       │
   Clinician query ──▶ Retrieval ──▶ Top-K chunks (redacted)
                                                       │
                                                       ▼
                                          Philter AI Proxy
                                          (defense-in-depth pass)
                                                       │
                                                       ▼
                                       LLM (self-hosted OR
                                       BAA-covered hosted)
                                                       │
                                                       ▼
                                       Response scanned for PHI
                                                       │
                                                       ▼
                                                  Clinician

      [All stages logged for audit; Phield monitors detection patterns]

Each of the seven exposure points from the previous section gets handled by a specific architectural decision. Walking through them:

Step 1: ingestion-time redaction with Safe Harbor policy

The first checkpoint. Every chart note that enters the pipeline passes through Philter with a HIPAA Safe Harbor policy applied. The 18 identifier categories under 45 CFR § 164.514(b)(2) are redacted using per-entity strategies appropriate for downstream use:

  • Patient names → replaced with consistent pseudonyms (Patient_A47 stays Patient_A47 across mentions for retrieval coherence)
  • Dates → shifted by a per-patient random offset (preserving intervals; obscuring absolute dates)
  • MRNs → hashed with a secret salt (joinable; not reversible)
  • Phone, email, address, SSN → redacted entirely
  • ZIPs → truncated to 3 digits where population > 20,000
  • Provider names, organization names → replaced with consistent pseudonyms

The full Safe Harbor identifier mapping covers each in detail. The output of this step is text that is no longer PHI under HIPAA — safe to send downstream.

Step 2: self-hosted embedding

Even with the text redacted, sending it to a hosted embedding API is unnecessary risk. Use a self-hosted embedding model — Sentence-Transformers (all-MiniLM-L6-v2, BGE-M3, etc.) on your own infrastructure, or BedRock if you have an AWS BAA. The embedding step happens entirely inside your perimeter.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("BAAI/bge-large-en-v1.5")
embeddings = embedder.encode(redacted_chunks)

For very high-volume workloads, a small inference-server deployment (Triton, vLLM, TGI) running the embedding model with batching and GPU acceleration is straightforward and keeps the data path inside your VPC.

Step 3: self-hosted vector store with BAA-compliant storage

Three options for the vector store, in increasing complexity:

  • pgvector on existing Postgres. If you already have a HIPAA-compliant Postgres, adding the pgvector extension is the lowest-friction option. Encryption at rest, IAM, audit logging all inherit from existing controls.
  • Self-hosted Chroma / Qdrant / Milvus. Drop-in deployments in your VPC. Use the cloud provider's encrypted volumes; back up to BAA-covered S3.
  • OpenSearch with the vector engine. If you're already running OpenSearch for log analytics, the k-NN plugin gives you vector search without standing up another piece of infrastructure.

The non-negotiable: the vector store must be inside the same BAA-covered infrastructure as the rest of the pipeline. Embeddings are partially invertible; hosted vector providers don't typically sign BAAs.

Step 4: retrieval, then defense-in-depth at inference

Retrieval is unchanged from a generic RAG — the user query (also passed through Philter to redact any PHI in the query itself) is embedded and used to retrieve the top-K chunks. The chunks are already redacted from step 1.

At inference time, the retrieved chunks plus the query plus the system prompt get assembled and sent through the Philter AI Proxy on the way to the LLM. Even if ingestion-time redaction had a hole — a novel entity type, a custom identifier the policy didn't anticipate — the proxy redacts before forwarding. The application code keeps calling its existing OpenAI / Anthropic / Bedrock client; only the base URL changes.

# Application code stays the same; just route through the proxy
client = OpenAI(
    api_key=KEY,
    base_url="https://philter-proxy.internal/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",  # or self-hosted; see below
    messages=[
        {"role": "system", "content": "You are a medical reference assistant..."},
        {"role": "user", "content": assemble_prompt(query, retrieved_chunks)},
    ],
)

The proxy also scans the response on the way back. If the model hallucinates PHI — not uncommon with summarization tasks — the scan catches it before it reaches the user.

Step 5: the LLM provider question

Two paths, with sharply different operational profiles:

  • BAA-covered hosted LLM. AWS Bedrock with a Business Associate Addendum, Azure OpenAI in HIPAA-eligible regions, or Google Vertex AI with an executed BAA. Higher per-call cost, lower operational burden. The redaction work above means the BAA is your backstop, not your only defense.
  • Self-hosted open-weights LLM. Llama 3 70B, Mistral, Qwen, etc. running on your own GPUs (or via a BAA-covered managed inference service like Together AI's HIPAA-eligible endpoints). No external network hop at inference time; no provider-side logs; maximum control.

For high-volume internal-tool deployments, self-hosted is usually the right answer once you're past the prototype phase. For pilot deployments or low-volume workloads, BAA-covered hosted is faster to ship.

Step 6: audit logging

The thing OCR will ask to see during any compliance review: what queries hit what data, when, by whom?

  • Log every ingestion event: which source document, what time, what entity types were detected and redacted.
  • Log every retrieval query: who asked, what was retrieved, when. Don't log the raw query if it's been sanitized; log the sanitized version.
  • Log every Philter AI Proxy interaction: what entity types were redacted, in which direction.
  • Log every model response: a hash of the response, the entity types caught by the outbound scan.

Phield handles the live monitoring side — tracking detection volumes, flagging anomalies (e.g., an upstream chart-format change that's causing the redactor to miss MRNs).

Step 7: validate and re-validate the policy

The policy file is not "set and forget." Run Philter Scope against a gold-standard set of representative chart notes as part of CI:

  • Recall must stay ≥ 99% for HIPAA Safe Harbor identifiers. Anything less is a breach in waiting.
  • Precision should stay ≥ 90% to keep the de-identified data useful for retrieval.
  • Per-entity-type breakdowns catch when a specific identifier type (e.g., a new payer's beneficiary number format) starts being missed.

The CI integration is the same compliance-as-code pattern we use for general redaction policies — specialized for the Safe Harbor thresholds.

Conversation memory: a special note

Medical chatbots are conversational; users ask follow-up questions. Without explicit handling, conversation history accumulates PHI exposure with every turn. Three rules:

  • Redact every turn before it gets added to history, not just the most recent.
  • Summarize older turns instead of replaying them verbatim. A 200-token summary of the conversation so far is safer than 20 full turns.
  • Bound history length explicitly. Drop turns older than N from the prompt context.

See "Prompt Engineering for Privacy" for the full pattern set.

The BAA chain

HIPAA requires a Business Associate Addendum with any vendor that touches PHI. In this architecture, the candidate vendors are:

  • Your cloud provider (AWS / Azure / GCP) — standard BAA, well-documented.
  • The LLM provider (if you went hosted, not self-hosted) — BAA available with AWS Bedrock, Azure OpenAI, Google Vertex. Not available with OpenAI's direct API (as of writing); not available with Anthropic's direct API.
  • The vector store provider (if hosted, not self-hosted) — rare; this is one of the strongest arguments for self-hosting.
  • The embedding provider (if hosted, not self-hosted) — uncommon; self-hosted is the standard answer.

The architecture above is designed to minimize the BAA chain — ideally to just your cloud provider (everything else runs inside your VPC) plus optionally a BAA-covered hosted LLM.

The bottom line

A HIPAA-compliant medical chatbot isn't a "RAG chatbot with extra steps." It's a different architecture entirely — one where redaction happens at three points (ingestion, query, inference), the data path stays inside BAA-covered infrastructure, every interaction is audited, and the policy is continuously validated against precision/recall thresholds.

The good news: the toolkit to do this is mature. Philter for redaction, Philter AI Proxy for the inference defense layer, Philter Scope for ongoing validation, Phield for monitoring — all open source, all self-hosted, all sharing one policy format. No proprietary stack, no vendor lock-in, no SaaS data path.

If you're building this for a health system or healthtech and want help pressure-testing the architecture against your specific compliance posture, get in touch. We've done it for clinical-summarization tools, internal-docs chatbots over clinical guidelines, and patient-facing FAQ assistants — the patterns are stable; the specifics of your data and your IT environment are where the consulting work concentrates.