Project case study · Research software · Split integrity · Reproducibility
Holdout Ledger: inspect train/test overlap with reviewable evidence
Data leakage can conceal itself behind apparently distinct record IDs. Holdout Ledger lets an engineer open a small local example, inspect the witnesses behind cross-split matches, review a proposed repair, and save a separate provenance-bearing result.
01 · Context
Research question
An aggregate duplicate count does not explain which exact records cross the evaluation boundary, how near-copies or groups connect, or whether a repair safely preserves held-out examples.
02 · Methods
Approach
- Builds bounded, explainable normalized-exact and lexical near-copy evidence, along with group-ID overlap checks.
- Represents connected components with source-split membership and individual match witnesses.
- Provides local CLI, a FastAPI-backed evidence explorer, offline HTML reports, and persisted SQLite audits.
- Generates source-fingerprint-bound plans that require human review, preserve test records, and write repaired data separately.
- Runs seeded synthetic controls and reproducibility-related CI checks; feedback is voluntary and must not include patient or proprietary data.
03 · Evaluation
Reported results
- The documented invented-records demo runs locally without a GPU, remote model API, account, or dataset download.
- GitHub Actions passed on the main branch on 2026-10-16; this verifies the configured checks, not field performance.
- Public onboarding includes a first-run walkthrough and optional privacy-safe tester feedback form.
04 · Scope
Limitations
Detection is restricted to normalized-exact text, lexical near matches, and declared entity groups. A cluster may be transitive; pairs within it need not directly match. Semantic paraphrases, missing group identity, unseen sources, and real-world leakage can escape the audit. No independent real-world sensitivity, clinical utility, or user-adoption figures are established.