Introducing Arbiter: Human-in-the-Loop PII Redaction

Automated PII redaction is genuinely great at most of the job. Pattern matching nails SSNs, credit cards, and phone numbers; PhEye's NLP models catch names, addresses, and organizations in unstructured text; Philter's policy engine ties it all together. For the majority of detections, the automated layer is faster, more consistent, and more comprehensive than human review.

And then there's the last few percent.

The detection that fires on the phrase "Huntington's disease" because "Huntington" is a name. The credit-card-shaped number that's actually a transaction ID. The patient name that's also a famous historical figure. The street address inside a quoted news article that's part of the public record. The mention of a person's name in a context where the policy says it shouldn't be redacted, or vice versa. These are the cases where automation can't decide on its own — and where ignoring the ambiguity means either over-redaction (data utility destroyed) or under-redaction (compliance failure).

Today we're releasing Arbiter — the open source human-in-the-loop review surface for redaction pipelines. Arbiter is the answer to "what happens to the cases the automated layer can't decide on its own?"

What Arbiter does

Arbiter sits between your automated redaction pipeline (typically Philter or the Phileas library) and your downstream consumers. Documents flow in; automated detection runs; results land in Arbiter's review queue; human reviewers work through the queue accepting, overriding, or exempting individual detections; the final document goes downstream with both the automated decisions and the human overrides applied.

The core review interface is built around three actions per detection:

• Accept. The automated decision was right. Redact this entity.
• Override. The automated decision was wrong — either a false positive ("this isn't actually PII") or a missed positive ("there's PII here the engine didn't flag").
• Exempt with a code. The automated decision was technically correct but doesn't apply here. Apply a structured exemption code with a reason — "public record," "scientific reference," "consented disclosure," whatever the policy taxonomy allows — that flows into the audit trail.

The reviewer never has to read full documents end-to-end. The queue surfaces only the spans that need human attention; everything the automated layer handled confidently flows through untouched.

Core features

Full-text search across the document corpus

Reviewers can search by content, by detected entity type, by reviewer status, or by any combination. Common workflow queries:

• "needs review and contains SSN" — today's work queue, scoped to high-priority entity types.
• "exempted last week" — quality-spot-check sample for the lead reviewer.
• "reviewed by alice and flagged for override" — consistency review across one reviewer's work.
• "contains 'consent' near person" — documents that may have consent-based exemptions.

The search runs over both document content and the structured metadata Arbiter captures (decisions, exemption codes, reviewer assignments, timestamps).

Structured exemption codes

Exemptions are first-class data in Arbiter, not free-text notes. Each policy defines a set of valid exemption codes — typically things like:

• PUBLIC_RECORD — the information is already in the public domain
• SCIENTIFIC_REFERENCE — a person's name used as the name of a disease, theorem, or scientific concept
• CONSENT_GIVEN — the data subject explicitly consented to disclosure
• HISTORICAL_FIGURE — a deceased public figure no longer subject to privacy protections
• BUSINESS_NAME_NOT_PII — an organization name the engine flagged as a person

When auditors ask "why didn't you redact this?", the answer isn't "the reviewer thought it was fine" — it's the specific exemption code applied, the reviewer who applied it, and the timestamp. Far more defensible.

Pluggable data sources

Arbiter imports documents from where your data already lives:

• OpenSearch and Elasticsearch. Pull documents directly from a search index using a configurable query. Useful when source documents are already indexed for other reasons (logs, archived correspondence, knowledge bases).
• RDBMS. Pull document content (and metadata) from PostgreSQL, MySQL, SQL Server, etc. via a configurable SQL query.
• Local files. Point Arbiter at a directory or list of files. Simple for one-shot batches and demo workloads.

The same review workflow operates regardless of source. Reviewers see documents and detections; they don't see (or care) where the underlying storage lives.

Reporting that compliance can use

Arbiter generates the reporting artifacts compliance teams ask for during audits:

• Per-reviewer throughput. How many documents has each reviewer worked through? How many decisions per hour? Trend over time.
• Per-entity-type exemption rates. What fraction of detected SSNs were exempted? Of person names? A high exemption rate may indicate the automated policy is too aggressive on that entity type.
• Per-document review status. Which documents are still in queue, in progress, completed, or escalated?
• Time-to-completion histograms. How long does a typical document take to review? Where are the outliers, and why?

Built on Philter, configurable everywhere

Arbiter doesn't reinvent detection — it uses the same Philter API and the same JSON policy files that power your automated pipeline. The entity types, confidence scores, and policy decisions Arbiter shows reviewers are the same ones Philter applies elsewhere in your stack. One configuration; two consumption surfaces (automated and human-in-the-loop).

Beyond detection, almost every aspect of Arbiter is configurable: reviewer roles and permissions, exemption-code taxonomies, queue prioritization rules, document import schedules, reporting destinations.

Where Arbiter fits in a redaction pipeline

  Source ──▶ Phinder ──▶ Philter ──▶ Arbiter ──▶ Final output
            (find)   (auto-redact) (human review)
                                       │
                                       ▼
                                 Philter Scope
                                 (measure policy
                                  against reviewer
                                  decisions)

The full lifecycle:

1. Discovery identifies where PII lives (Phinder or your existing inventory).
2. Automated redaction runs Philter against every document at scale, capturing detections and confidence scores.
3. Triage sends only the cases that need human review into Arbiter's queue — typically low-confidence detections, novel entity types, or policy-flagged ambiguities.
4. Human review in Arbiter applies accept / override / exempt decisions, captured with reviewer attribution and timestamps.
5. Policy improvement uses the reviewer decisions as feedback for Philter Scope — high override rates on a specific entity type indicate the automated policy needs tuning. The next policy iteration is informed by the human decisions, closing the loop.

Where Arbiter matters most

AI training data preparation

Training datasets used to fine-tune ML models are a particularly high-stakes case for PII redaction. A leaked patient name memorized into a model is functionally impossible to remove after the fact. The cost of a missed detection at training time is permanent.

Arbiter is the right tool for the final human pass on training-data candidates — the queue review that catches what the automated layer missed before the data is committed to a training run.

Healthcare de-identification

HIPAA Safe Harbor allows automated de-identification but doesn't excuse misses — "I deployed an automated tool" is not a defense if PHI leaks through. Arbiter provides the human-review layer that turns Safe Harbor automation into Safe Harbor compliance, with the per-document audit trail HHS expects.

The Safe Harbor blueprint we published earlier describes the architectural patterns; Arbiter is the piece that handles the case where the 18-identifier engine is ambiguous and someone has to decide.

Legal e-discovery

Legal redaction (privilege review, FRCP Rule 5.2 compliance, e-discovery production) has always required human judgment for the cases at the margin. Arbiter brings that human-judgment layer into the same architecture as the automated detection — reviewer decisions are first-class data, not loose notes in a separate system. The legal-vertical post covers the broader workflow context.

Everyday regulated workflows

Banks, insurance carriers, government FOIA-response teams, and education platforms all share the same pattern: automated redaction is fast and good but not perfect, and the cases at the margin are where compliance lives or dies. Arbiter is the surface that brings human reviewers into the loop without making them read every document end to end.

Getting started

Arbiter is Apache 2.0 open source. Three paths:

1. Source. Clone from github.com/philterd/arbiter. Build, configure, deploy.
2. Container. Pull the Docker image; deploy via Kubernetes, ECS, or whatever your container orchestration is.
3. Hosted via consulting. If you'd rather have us configure Arbiter against your existing Philter deployment, your data sources, and your reviewer workflow, get in touch. We've done this for healthcare and legal customers; the configuration is the part that takes domain expertise.

For full feature documentation, see the Arbiter product page or Arbiter docs.

The bottom line

Automated PII redaction is great. It is not, and shouldn't be expected to be, perfect. The cases at the margin — the ambiguous detections, the policy edge cases, the missed positives — need humans to decide, and the humans need a tool that doesn't force them to start from scratch.

Arbiter is that tool. Built on Philter, designed around the accept / override / exempt loop, configured to bring in data from wherever it lives, and reporting in the shape compliance teams actually use. Open source, self-hosted, and shipping today.

Start with the Arbiter product page, or jump straight to the repository.