sift

Sensitive-data discovery over files. Finds PII (SSNs, cards, emails, phones) and secrets across a directory, controls false positives with Benjamini–Hochberg false-discovery-rate control, and reports every file it could not read.

An information-theory measurement layer (Shannon entropy) feeds a detection-theory decision layer (FDR). The idea it carries: a scanner runs one hypothesis test per candidate — is this a real identifier, or a coincidental pattern match? — so it runs thousands of tests at once, and controlling the false-discovery rate per document is what separates real hits from the flood of look-alikes (a bare 9-digit number, a random Luhn-valid id, a git SHA).

Install

pip install -e .            # or: uv pip install -e .

Zero runtime dependencies. Python ≥ 3.11.

Use

scan a directory

python -m sift scan path/to/dir

sift — scan of path/to/dir  (FDR q=0.05)
  files: 3 read, 1 unreachable {'binary': 1}  |  4 total
  candidates: 54  ->  4 sensitive (FDR-controlled), 50 filtered as coincidental
  sensitive by type: {'credit_card': 1, 'email': 1, 'secret': 1, 'ssn': 1}
  findings (redacted, by strength):
    email        j***@example.com           p=0.0002     .../customer.txt
    secret       sk_l…(44 chars)            p=0.0004     .../config.env
    credit_card  **** **** **** 1111        p=0.0004     .../customer.txt
    ssn          ***-**-6789                p=0.001      .../customer.txt
  unreachable (1): binary — reported, not dropped

Values are masked in every output, including the --json report — a scanner should not print what it finds.

bench — the measured result

python -m sift bench

sift benchmark — 50 sensitive + 300 coincidences (synthetic, ground-truth)
  recall    0.8    (40 found, 10 missed — the missed are the bare, context-free ones, by design)
  precision 0.909
  false positives:  without FDR 199  ->  with FDR 4   (FDR removed 195 at q=0.05)

bench generates a labelled synthetic corpus — planted sensitive values plus coincidental look-alikes among benign filler — and scores sift against the ground truth. Fully synthetic; deterministic in --seed; nothing real. The number that carries the design: a plain pattern matcher would flag ~199 of the coincidences; FDR takes that to ~4. And recall is 0.80, not 1.0 — the missed items are the bare, context-free ones, which are indistinguishable from a coincidence and so are left unflagged rather than guessed.

How it works

files → detect candidates (sits) → measure (entropy) → decide (FDR) → verdict + coverage

• stats.py — Shannon entropy (IT) and Benjamini–Hochberg (DT), pure-Python reimplementations of the tested numpy versions in forge_core.
• sits.py — the detectors; each match carries a coincidence score = validator-rate × context-factor (used as an approximate p-value), so a bare Luhn-valid number is weak evidence and the same number beside "card number" is strong.
• scan.py — walk (pruning cache/dependency dirs, skipping known-binary extensions), detect, FDR-control per document, and record every unread file with the reason (binary / too-large / permission).
• generate.py / measure.py — the labelled corpus and the scoring behind bench.

Scope and limits

• A proof of concept — a small study applying an IT × DT detection battery to data at rest.
• The scores are hand-set structural estimates, not calibrated p-values, so the FDR bound is approximate, not exact. Why they can only be estimates: a real p-value is a tail probability under a null model — what coincidental matches look like in the data being scanned — and that null is corpus-specific (different data has different coincidences), so it has to be measured from a benign sample of the estate, not assumed. The hand-set scores stand in for it: they order candidates soundly, but ordering alone is not a valid FDR bound. So calibration is a per-deployment step, not a ship-once feature — the empirical null is built fresh against each target. This version uses the heuristic scores; the calibration machinery is prototyped separately.
• A practical source for that per-target null: other tools' false positives, once an analyst has dispositioned them, are already-labeled coincidences drawn from the real estate — so they could feed the empirical null directly, and every disposition sharpens it. A cheap, self-improving way to calibrate against a live target.
• Entropy detects secret-shaped tokens; it cannot separate a 9-digit SSN from a 9-character key on short strings, so structured PII is left to the validators.
• Context-free sensitive data is not recoverable by construction — that is the recall gap, and it is reported rather than papered over.
• Local filesystem only. Other sources (cloud stores, a DLP/classification API) would be adapters over the same core.

Lineage

The information-theory and detection-theory primitives are lifted from forge_core, an IT × DT detection battery (entropy, KL, mutual information; CFAR, CUSUM, FDR, matched filter). sift applies two of them to a data-security problem.

License

Apache-2.0 — see LICENSE and NOTICE.