Ability — Entity Resolution
Ability turns messy, overlapping records into one clean, traceable set of entities. It works in five stages and never destroys source data: every merged record keeps links back to the originals it came from.
Record linkage is the matching math (does record A match record B?). Entity resolution is the whole job: matching, clustering matches into entities, scoring confidence, and keeping provenance. Do the whole job.
Before starting: the alignment check
Messy requests are normal. Read typos and shorthand for intent and never comment on spelling. Before real work, state three short lines so the user can correct course cheaply:
SAID: <what the user literally asked>
INFERRED: <the actual problem to solve>
ASSUMED: <the main assumption you are making — ask at most one question if truly blocked>
Skip this for quick questions; use it when you are about to process data or design a pipeline.
The 5-stage pipeline
1. Profile
Look at every field before matching anything. Report per field: null rate, distinct count, top values, format patterns, and encoding problems (UTF-8 vs Latin-1). Profiling tells you which fields are trustworthy enough to match on and which values are so common (e.g. "Smith", "123 Main St") that agreement means little.
2. Standardize
Normalize before comparing, or the scores will be wrong: lowercase, strip punctuation, expand abbreviations (St → Street), parse names into parts, normalize phone numbers to digits, normalize company suffixes (Inc., Ltd, LLC), and standardize addresses (postal verification where available).
Messy real data almost always also needs:
- Nickname table (Bob/Robert, Peggy/Margaret, Sasha/Alexander, Trish/Patricia) and initials ("J.") treated as compatible with any name starting with that letter.
- Swapped name fields — also compare first↔last reversed.
- Ambiguous dates — "04/05/1990" could be April 5 or May 4. Keep both readings and treat a day/month swap as a partial match, not a conflict.
3. Block
Comparing every record to every other is N² — 1M records means ~500 billion pairs. Blocking only compares records that share a key. Use several passes with different keys so a typo in one field does not hide a true match:
- Phonetic surname (Soundex / NYSIIS) + birth year
- Postcode prefix + house number
- Last 4 of an ID + ZIP
- Phone or email
- For text-heavy data: MinHash LSH on token sets, or embeddings + nearest-neighbour search (FAISS/HNSW)
Measure it:
- Reduction ratio = 1 − (pairs compared ÷ all possible pairs) — aim for > 99%
- Pair completeness = true matches that land in some block ÷ all true matches — aim for ≥ 95%
4. Score — the 3-step cascade
Run the cheap, certain checks first and the expensive ones only where needed.
- Deterministic: a reliable unique ID agrees (national ID, tax ID, account number, verified email) → match.
- Fuzzy: no ID match → compare fields with the right algorithm per field (see table) and combine them with Fellegi-Sunter weights.
- Machine learning / human review: still ambiguous → a classifier (e.g. logistic regression on the field-similarity features) if labeled data exists, otherwise route to a human review queue.
| Field type | Use |
|---|---|
| Person names | Jaro-Winkler (rewards matching prefixes), plus nickname lists |
| Company names | Token sort ratio (word order doesn't matter) after suffix cleanup |
| Addresses | Levenshtein after standardization; house and unit numbers must agree exactly |
| IDs, phone, email | Exact after normalization |
| Long text | Jaccard / cosine on tokens |
Fellegi-Sunter in brief — for each field k:
- m = P(field agrees | records truly match) — how reliable the field is
- u = P(field agrees | records don't match) — how often it agrees by chance
- agree weight = log2(m/u); disagree weight = log2((1−m)/(1−u))
- total score = sum of the weights for each field's outcome
Rare values carry more evidence: agreeing on "Lebowski" (tiny u) is far stronger than agreeing on "Smith". Use value-specific u where you can. In practice: a rare full name plus an exact birth date is strong evidence even when address, phone and email all changed (the person moved); a common name plus birth date is not.
To score pairs, run scripts/fs_score.py instead of computing by hand — it applies
the weights, converts the score to a match probability given prior odds, and assigns
the decision band. Run python scripts/fs_score.py --help for usage.
Full formulas, the worked example, and m/u estimation (including EM) are in
references/methods.md; read it when the user wants the math or needs to tune weights.
Two traps the score alone won't catch:
- Households. Twins, spouses, and roommates share surname, address, and sometimes birth date, so their score can look like a near-certain match. If first names differ (and aren't nicknames of each other) and personal contacts (email, phone) conflict, keep them apart or send them to review — whatever the score says.
- Placeholder values. Dates like 2000-01-01 or 1900-01-01, phones like 555-0000, and emails like [email protected] are filler, not evidence. Profiling (stage 1) should flag values that repeat suspiciously; give them no weight.
Decision bands (tune on labeled data; these are starting points):
| Match probability | Action |
|---|---|
| ≥ 0.95 | Auto-match |
| 0.80 – 0.94 | Human review queue |
| < 0.80 | Non-match |
5. Merge — survivorship and golden records
Group matched pairs into entities (connected components / graph clustering — watch
for chains where AB and BC but A≁C; split those for review). For each entity build
one golden record by choosing each field with a stated rule:
- Most recent (latest timestamp) — for addresses, phone
- Most complete (non-null, longest) — for names, descriptions
- Trusted source (system ranking) — for IDs, legal names
Never overwrite or delete originals. Store: golden record ID → list of source record IDs, the rule that picked each field, and the match score that joined them.
Validating results
Ask for, or help build, a small labeled set (a few hundred pairs marked match/non-match). Report precision, recall, and F1 on it, plus reduction ratio and pair completeness for blocking. Thresholds without validation are guesses — say so when there is no labeled set.
Evidence tags
When claiming a result, say how you know it:
- [DOC] — from a spec or documentation, not run
- [RAN] — executed in this session
- [PASS] — checked against labeled/test data
Checkpoint review
On long jobs (roughly every five working turns, when the user reports a mistake, or when they type "checkpoint" or "dream"): pause the project work and briefly review your own recent turns. List each mistake or drift from the user's stated constraints, its cause, and the fix you will apply going forward. Then resume. Keep the review out of deliverables — it is for the conversation, not the user's data or documents.
Output for a matching job
End a completed job with:
ENTITIES: <input records> → <resolved entities> (<duplicates found>)
BANDS: auto <n> | review <n> | non-match <n>
QUALITY: precision <x> | recall <x> | F1 <x> (or "not validated — no labeled set")
FILES: <where the golden records, crosswalk, and review queue were saved>
Privacy
Matching often involves personal data. Process it locally, don't paste identifiers
into URLs or external services, and when linking across organizations suggest
privacy-preserving linkage (Bloom-filter encoding of names, compared with Dice
similarity) — details in references/methods.md.