Reproducible benchmarks
Same test set, same metrics, every run. Two engineers comparing two policies see the same numbers. No more debates about whether the new rules are actually better.
Score your redaction policies
Philter Scope
Philter Scope is a standalone audit tool that scores redaction policies against gold-standard test data. Stop guessing whether a policy change made the pipeline better. Measure it, version it, and fail the build when it regresses.
Score a policy against gold-standard data
Philter Scope compares a policy's redacted output against your labeled gold-standard set and reports precision, recall, and F1 for every entity type, so a change that regresses recall is caught before it reaches production.
scoring healthcare-v7.json · golden set: clinical-notes-500
EntityPrecisionRecallF1
SSN
1.00
1.00
1.00
PHONE
0.99
0.98
0.98
PERSON
0.97
0.95
0.96
DATE
0.94
0.88
0.91
MRN12 of 43 missed
0.96
0.72
0.82
Overall F1 0.94. Recall on MRN fell to 0.72, below the 0.75 threshold, the kind of regression Philter Scope fails a build on.
Run it with Docker
Philter Scope ships as a multi-arch image (x86-64 and arm64) on Docker Hub at philterd/philter-scope. The repository ships a docker-compose.yaml that scores a policy and then serves the report in your browser.
# 1. Grab the Compose file
curl -O https://raw.githubusercontent.com/philterd/philter-scope/main/docker-compose.yaml
# 2. Put your labeled gold standard in ./data/golden and the text
# you want scored in ./data/raw
mkdir -p data/golden data/raw
# 3. Score the policy against your Philter, then open http://localhost:5000
export PHILTER_URL=http://philter.internal:8080
docker compose up
The audit writes its reports into ./data, then the dashboard serves them. The running guide covers every flag, both golden-data formats, and the MongoDB setup that adds shared history and the trend chart below.
Why Philter Scope
Gold-standard comparison
Annotate a representative sample of your real text once. Philter Scope compares any policy output against that ground truth and reports precision, recall, and F1 per entity type, along with an entity type confusion matrix showing where detectors misclassify.
Per-entity breakdown
Aggregate scores hide problems. Philter Scope reports per-entity-type metrics so you can see exactly which detectors are weakest and where the next tuning pass should land.
CI integration
Run it as a step in your CI pipeline. Fail the build when precision or recall regresses below a threshold; catch policy regressions before they reach production.
Audit artifact
The evaluation report is the artifact regulators and auditors actually want to see. Demonstrate that your redaction pipeline is verifiably correct, not just "trust us, it works."
Open source
Pair with Phileas and Philter, or use against any redaction output. The evaluation logic is open: your QA team can read every line of the code that scores them.
See it in action
The Philter Scope dashboard breaks down precision, recall, and F1 by entity type, so you can see exactly where your policy is strong and where it needs tuning.
Frequently asked questions
If something here isn’t covered, get in touch and we’ll answer.
What is Philter Scope?
Philter Scope is a standalone audit tool that scores redaction policies against gold-standard test data. Instead of guessing whether a policy change made your pipeline better, you measure it: Philter Scope reports precision, recall, and F1, per entity type, so you can version a policy and fail the build when it regresses.
Why does measuring redaction matter?
Redaction feels binary, but it is statistical. Every policy decides, entity by entity, what to catch and what to let through, so changing a rule, adding a detector, or swapping in a model places a bet on thousands of decisions nobody will read by hand. A spot check of a few documents that looked clean is a hope, not an audit artifact. Measuring turns the policies you run through Phileas and Philter into something you can version, review, and verify, and it gives an auditor a reproducible report scored against ground truth rather than an assurance that the pipeline works. For a deeper walk through the three metrics and how to read them, see Privacy shouldn't be a guessing game.
What do precision, recall, and F1 actually tell me?
Precision is how much of what you redacted was genuinely sensitive, so it tells you where the policy is destroying useful data by over-redacting. Recall is how much of the real PII you actually caught, so it tells you exactly where data is leaking. F1 combines the two into a single score. Philter Scope reports all three per entity type, because an aggregate number can read 98 percent while quietly missing most medical record numbers.
Do I need labeled data to use it?
Yes. You annotate a representative sample of your real text once to create a gold-standard ("golden") set. Philter Scope then compares any policy's output against that ground truth and produces the scores, along with a confusion matrix showing where detectors misclassify one entity type as another.
Can I run Philter Scope in CI?
Yes, and that is the point of it. Run it as a CI step, set a floor on precision or recall for the entity types you care about, and fail the build when a policy change regresses below the threshold. A regression that re-exposes Social Security numbers gets caught the same way a broken unit test would, before it reaches production.
Does Philter Scope only work with Philter?
Is Philter Scope open source?
Yes. Philter Scope is open source under the Apache License, version 2, and the code is on GitHub.
Ready to use Philter Scope?
Grab the open source and run it yourself, or work with our team directly. Pick the path that fits.