Synthetic Data Overlay

Synthetic Data Overlay

Replace sensitive values with synthetic ones that preserve shape and joinability.

When to invoke

• pii-flag has produced a report and the user wants remediation by substitution (rather than drop or hash).
• User wants to share or publish a dataset while preserving analytical structure.

Strategies

Choose per column based on the kind of data:

| Category | Strategy | |---|---| | Names | Generate synthetic names via Faker (locale-matched if possible). | | Emails | Synthetic email using generated name + fake domain (@example.com). | | Phone numbers | Faker generator with country locale matching the original. | | Addresses | Faker address; preserve country/city-level granularity if needed for analysis. | | Dates of birth | Shift by a random offset (per-row or constant), OR bucket to age range and resample. | | Government IDs | Generate format-valid but non-real IDs (e.g. valid checksum, clearly-fake prefix). | | Credit cards | Use PAN test ranges (e.g. 4111-1111-...) which are Luhn-valid but publicly known as test numbers. | | Geocoordinates | Add uniform noise within a radius (e.g. ±500m) OR snap to city centroid. | | Free text with PII | Presidio-anonymizer with per-entity strategies. | | Arbitrary categorical | Sample from original distribution (preserves frequencies) or a uniform placeholder. | | Numerical (non-PII but sensitive) | Add calibrated noise or use synthetic generation libraries (sdv, synthcity). |

Procedure

1. Require a PII report as input (from pii-flag). Do not re-detect from scratch here.
2. Present the strategy plan per flagged column, including a preview of 3 synthetic replacements alongside the originals (masked). Confirm with the user before applying.
3. Preserve referential integrity:

• Same original value → same synthetic value across rows AND across related files (use a deterministic seeded mapping, keep the mapping out of the output).
• Cross-column consistency (name ↔ email ↔ phone): generate as a coherent persona, not independently.

1. Apply replacements and write the overlay output with _synthetic suffix. The original file is never modified.
2. Destroy the mapping between real and synthetic values by default, OR store it encrypted outside the dataset folder with a user-confirmed path. Never commit the mapping to the dataset's git repo.
3. Re-run pii-flag on the output as a verification step. Report residual findings.
4. Update the data dictionary — record which columns were synthesised and the strategy used. Mark the dataset as "synthetic overlay applied" in provenance.
5. Refuse to push synthetic datasets as real — if the user subsequently invokes hf-dataset-push, ensure the dataset card marks the data as synthetic-overlaid with a clear disclaimer.

Dependencies

pip install pandas faker
# optional
pip install presidio-anonymizer sdv synthcity

Edge cases

• Statistical fidelity matters — if the downstream use is modelling, random sampling from a uniform distribution will distort results. Prefer distribution-preserving methods (sdv CTGAN, sampling from the original distribution) and document the chosen approach.
• Locale mismatch — don't generate US-style phone numbers for Israeli data. Use locale-aware Faker instances per row.
• Re-identification risk — even synthetic overlays can leak information if quasi-identifiers (age + city + profession) uniquely identify individuals. Warn the user and recommend k-anonymity checks for high-risk datasets.
• Free-text residue — PII in long-form text is hard to fully scrub; note residual risk explicitly.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: danielrosehill
• Source: danielrosehill/Claude-Data-Wrangler-plugin
• License: MIT

Install and usage instructions live in the source repository linked above.