Data Identity Resolution

Data identity resolution — attach, don't guess

data-semantic-quality owns the rules: trust ladder, confidence as a gate, cohort fences. This skill owns the procedure for linking a messy incoming record to a canonical entity key without inventing a second identity system.

No product match thresholds, no catalog taxonomies. Those stay in the product repo.

When to invoke

• Parsing free text / third-party ids onto a canonical entity key.
• NULL or unresolved identity keys growing day over day.
• Remapping / restamping historical links after a resolver change.
• Strong structured evidence disagrees with a weak title/heuristic.

Don't use

| Task | Go here instead | |---|---| | Quality flags, outliers, golden entity packs | data-semantic-quality | | Scoring published estimates vs later facts | data-product-eval | | Schema, watermarks, publish/OCC | data-apache-lakehouse / data hub | | Serving honors stored keys; does not re-resolve | data-api | | “Should this table exist?” | data-table-lifecycle | | Concrete matcher constants / synonym lists | product-repo skill |

Non-negotiables

1. No domain product nouns or concrete match thresholds in advice from this skill.
2. Ladder rules live in data-semantic-quality. Do not fork a second ladder here.
3. Unresolved is a first-class state, not a blank to be filled by the next weak hint.
4. Remap cannot undo a stronger fence. Restamp is ordered.
5. Low confidence never enters identity-assuming aggregates.

Principles

Each has a falsifiable Test:. Ladder mechanics live in data-semantic-quality (trust ladder + confidence gates). Workflow detail: [references/attach-workflow.md](references/attach-workflow.md).

1. Evidence before assignment

Collect ordered evidence (strong structured → catalog default → weak text) then decide assign / quarantine / leave unresolved. Do not assign inside the first matching if.

Test: invent a row where weak text and a strong structured field disagree. Does the weak path still win if you reorder branches?

2. Fail closed

Below the product’s confidence threshold the key stays null/unresolved and the row is excluded from identity-assuming rollups. “Average it in at a discount” is not fail-closed.

Test: can a below-threshold row appear in the primary aggregate without an explicit override flag?

3. Conflict quarantines

Strong-vs-weak mismatch is a quarantine (or a conflict label), not a coin flip. Record which evidence class produced the decision.

Test: is there a durable conflict/quarantine state a later job can count, or does the row silently pick a side?

4. Unresolved accumulation is a metric

NULL / unresolved keys are measured debt: count per source per day, with a rate (unresolved / landed). A resolver that “works” while the null pile grows is losing.

Test: can you plot unresolved-landed per source for the last N days from published tables, without reading code? If the rate is up and no one is paging, the metric is missing.

5. Remap is ordered and one-way through stronger fences

A restamp / re-resolve pass applies weaker evidence only where stronger evidence is absent. It must not clear a strong assignment or a quarantine to please a new heuristic.

Test: run restamp on a fixture that has a strong key and a tempting weak hint the other way. Does the strong key survive?

6. Backfill is a bounded, resumable lane

Unresolved backfill is not the incremental attach path. It has its own watermark or keyset, chunk size, and idempotent write. Hub principle 2 (watermark, not full recompute) applies — a crash must not restart the whole historical pile.

Test: kill a backfill at 40%. Does retry resume, or re-scan from record one and rewrite already-good keys?

7. One writer for the link

The attach job (or documented reconcile) is the only writer of the canonical key + confidence + evidence class. APIs, UIs, and rollups consume. A serving path that “fixes” identity at request time is a second resolver.

Test: change the attach threshold in one place. Do all consumers move on the next publish, or does one surface still re-parse text?

8. Coverage audit names identity rate

When someone asks “how much data do we have,” unresolved rate is part of the answer — row count alone lies. The data hub coverage-audit playbook includes this slice; this skill defines what “resolved” means (key present and above gate).

Test: a source with 1M rows and 85% null keys — does the audit report “1M rows” or “150k resolved”?

Workflow: attach or remap

Identity attach / remap:
- [ ] 1. Name the canonical key + what “unresolved” looks like on the row
- [ ] 2. List evidence classes in ladder order (product repo supplies the actual signals)
- [ ] 3. Assign only from the strongest available non-conflicting class
- [ ] 4. Conflict → quarantine; below threshold → unresolved (not a guess)
- [ ] 5. Persist key, confidence, evidence class, policy version
- [ ] 6. Incremental path is watermarked; historical remap is a separate resumable lane
- [ ] 7. Restamp fixture: strong assignment survives a contradictory weak hint
- [ ] 8. Publish unresolved-rate metric per source
- [ ] 9. Serving / rollups honor stored keys — no request-time re-parse
- [ ] 10. No domain thresholds committed into this pack

Common mistakes

• First matching heuristic writes the key.
• Weak title/lane stamp overrides a structured identifier because it ran first.
• NULL keys treated as “not a problem until someone searches that row.”
• Restamp job that reapplies the full matcher and clears quarantines.
• Historical remap on the incremental timer (O(history), no resume).
• API re-parses free text because “the lake key looks wrong.”
• Quoting source volume without the resolved rate.

References

• Attach / remap / backfill procedure: [references/attach-workflow.md](references/attach-workflow.md)
• Trust ladder + confidence-as-gate → data-semantic-quality (references/rule-scoping.md)
• Estimate vs later truth → data-product-eval
• Coverage audit (query prod, include identity rate) → data hub
• Serving consumes stored keys → data-api
• Incremental vs rebuild lanes → data hub principle 2

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Evan-Kim2028
• Source: Evan-Kim2028/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.