What is entity resolution?#

Deciding which rows in your data describe the same real-world thing.

You have a table. Somewhere in it, the same person appears three times — once as “Michaela Neumann”, once as “Michela Neuman”, once with a different address because she moved. There is no column that says these are one person, because if there were, you would not have a problem.

Entity resolution is the work of adding that column.

Every term this survey uses, before it uses it#

Record. One row. A person, a company, an object, a transaction — described by several fields.

Entity. The real thing a record describes. Three records, one entity.

Deduplication. Finding repeated entities within one table.

Linkage. Matching records across two tables. Same problem, different shape: in linkage you often know each record on the left matches at most one on the right, and that constraint is useful information.

Pair. Two records being considered together. The unit almost everything in this category operates on.

Candidate pair. A pair the system decided was worth comparing at all. See blocking.

Blocking. Choosing which pairs to compare. Necessary because comparing every pair is quadratic: 5,000 records make 12.5 million pairs, and 100,000 make five billion. A blocking rule says something like “only compare records that share a date of birth”, which throws away 99.9% of the work.

Blocking recall. The fraction of truly-matching pairs that survived blocking. This is a ceiling: a true pair the blocker never proposed cannot be recovered by anything downstream, no matter how good.

Reduction ratio. How much work blocking avoided. Usually 0.999-something, and usually the less interesting of the two numbers.

Comparison vector. For one candidate pair, a list of numbers — one per field — saying how much that field agreed. [0.95, 1.0, 0.0, 0.7] might mean the names nearly matched, the birth dates matched exactly, the postcodes did not match at all, and the addresses partly did.

Matching / scoring. Turning a comparison vector into one number: how likely is it that this pair is the same entity.

Threshold. The number above which a pair is accepted as a match. Somebody has to choose it.

Clustering. Turning accepted pairs into entities. This step exists because pairs do not compose cleanly — see below.

Transitive closure / connected components. The usual clustering method: if A matches B and B matches C, put A, B and C together, whether or not A and C were ever compared.

Canonicalization. Producing one surviving record per entity. Which of the three spellings goes into the final table, and which address.

Golden record / master record. The surviving record. Same idea, different vocabulary — the enterprise term is master data management.

Precision. Of the pairs you said were matches, what fraction really were. High precision means you did not merge things that should have stayed apart.

Recall. Of the pairs that really were matches, what fraction you found. High recall means you did not miss duplicates.

F1. The harmonic mean of precision and recall — one number, used because it punishes being very bad at either.

Adjusted Rand Index (ARI). A score for how well your grouping matches the true grouping. 1.0 is perfect; around 0 is what random guessing gets.

Ground truth. A dataset where somebody already knows the right answer, so a method can be scored instead of admired.

The problem, in one example#

Two rows:

Same person. The names differ by two characters, and everything else agrees.

Now two more:

Different people, and the names match exactly.

That is the whole difficulty. A string comparison says A and B are 90% similar and C and D are 100% similar, and gets both cases backwards. Entity resolution is the discipline of combining evidence across fields — and of knowing that “Smith” is weak evidence and “Ntsekhe” is strong evidence, because one is common and the other is not.

Why the pieces are in that order#

The four steps exist because each solves a problem the previous one created.

Blocking exists because comparison is quadratic. You cannot compare everything, so you compare a promising subset.

Scoring exists because agreement is not binary. “Michaela” versus “Michela” is neither a match nor a non-match; it is evidence, and it has to be weighed against the other fields.

Deciding exists because eventually somebody needs a yes or a no. A score of 0.83 is not an answer you can write into a table.

Clustering exists because pairs do not compose. This is the step people are most surprised by, and it deserves its own explanation.

Why pairs are not enough#

Suppose your system accepts three pairs: AB, BC, and A~D.

How many people is that? You cannot answer from the pairs alone. If you take the transitive closure you get one group of four — A, B, C and D — even though A and C were never compared and might be obviously different.

That transitivity is the trap. One wrong pair merges two entire entities. If B~C was a mistake, and B belonged to a group of three while C belonged to a group of four, you now have one wrong cluster of seven where there were two correct clusters. The pairwise score records that as one error among thousands. The entity list records it as two entities destroyed.

The reverse mistake is gentler. A missed pair splits one entity into two — one wrong entity, not two.

Over-merging is catastrophic and under-merging is merely disappointing. That asymmetry is why this category advises tightening rather than loosening, which is the opposite of the instinct that says “we want to catch all the duplicates”.

Two numbers, and the one people report#

Every run produces a pairwise score and an entity-level score, and they say different things.

From this survey’s measurements, on 5,000 records describing 2,000 real people:

The middle row is the one to look at. Pairwise F1 of 0.815 sounds like a working system. The same run found 2,511 people where there are 2,000, and got 27% of them wrong.

The top row is worse and looks better in the way that matters most to a reviewer: exact matching had perfect precision. It never merged anything wrongly. Anyone checking the merges finds no errors — because all of the error is in the 1,682 extra people it invented by failing to merge, and nobody inspects things that did not happen.

Two ideas that make the good tools good#

Fellegi-Sunter (1969). The statistical model underneath most of this category. For each field it estimates two probabilities: how often that field agrees when a pair is a match, and how often it agrees when the pair is not. The ratio between them is how much that field’s agreement is worth. It is why a matching date of birth counts for more than a matching first name, without anybody deciding that by hand.

Term-frequency adjustment. The same idea applied to individual values. Two people both named “Smith” is weak evidence; two people both named “Ntsekhe” is strong evidence. A hand-written score cannot express this, because it depends on how common each value is in your actual data. This is the single largest thing a probabilistic engine adds over a threshold you chose yourself.

Where entity resolution is not the answer#

When you just need to clean up one column. A list of supplier names with four spellings each is a text-normalization job. Reach for a tool that clusters values with a person confirming each merge.

When the data has a real identifier and it is merely damaged. Repairing formats and joining on the repaired key beats any matcher, and it produces a result you can explain.

When the ambiguity is definitional rather than technical. If nobody has decided whether a photograph and its negative are one object or two, no algorithm will decide it for you.

When somebody will inspect every merge anyway. Then you do not need accuracy, you need good ranked suggestions with the evidence visible — a different requirement, met by different tools.

The one-sentence version#

Entity resolution is four steps — decide what to compare, score it, accept or reject it, and group the accepted pairs into things — and the last step is where a single mistake costs double, which is why the number usually reported is not the number that matters.

S1 approach#

The boundary test#

Everything in this survey is sorted by one question, asked of the object a library hands back:

Does it tell you which records are the same entity, or only how alike two strings are?

That line matters because the two sides look identical in a feature list and are not the same product. A similarity function takes two values and returns a number. An entity resolver takes a table and returns a smaller table, where every input row carries the identifier of the thing it describes.

Between those two objects sit three steps, and a candidate is placed by how many it performs:

1. Blocking — deciding which pairs are worth comparing at all. Five thousand records make 12.5 million pairs; a hundred thousand make five billion. Every tool that survives contact with real data has an answer here, and the answer sets a ceiling nothing downstream can raise: a true pair the blocker never proposed cannot be recovered by any model.
2. Deciding — converting a score into a yes or a no. Somebody must choose a threshold. The candidates differ in whether that somebody is the library, the library’s training procedure, or the reader with a histogram and an afternoon.
3. Clustering — turning accepted pairs into entities. This is where the field’s arithmetic stops being intuitive: if A matches B and B matches C, but A does not match C, a pairwise answer is silent about how many people are in the room. Someone has to break the tie, and if the library does not, the caller does — usually by accident, and usually by taking a connected component and hoping.

Four capabilities are recorded rather than three, because a fourth appears once a category is examined and is missing from nearly all of it: canonicalization, producing one surviving record per entity. A cluster of six spellings is a question, not an answer, for anyone who has to write one row back into a system.

What counts as in scope#

A candidate is in scope if it takes records with several fields and produces a judgment about which records describe the same real-world thing. Requires all three:

• The unit is a record, not a string. A library that compares two names and stops is a primitive, and this corpus already covers those.
• The comparison is across fields with different reliability. A birth date is worth more than a first name; a shared surname in a village means less than it does in a country. Handling that asymmetry is the discipline.
• The output is about identity, not ranking. A search engine returns the best matches for a query. A resolver asserts that two rows are one thing, and takes responsibility for the assertion.

Deliberately included as scored candidates, not as controls:

• The database. An exact match on a normalized key, and a trigram similarity join in SQL, are what most teams actually ship, and are the thing every library on this list has to beat. They are scored on the same benchmark as everything else.
• A desktop application. OpenRefine is not a library and cannot be imported, and it is the correct answer for a whole class of reader. Excluding it on the grounds of packaging would be excluding it on the grounds that it is convenient to exclude.
• A commercial engine. Senzing’s SDK carries an open-source license and its engine does not. That combination is a fact a reader needs before, not after, a proof of concept.

Out of scope, and where it lives instead:

• Pure string metrics and phonetic algorithms — 1.032, 1.002, 1.030, 1.013.
• Approximate nearest-neighbor search over embeddings as a retrieval problem — 1.009, 1.203. It appears here only as a blocking strategy.
• Address and name parsing — a genuine dependency of this work, covered as a supporting tier rather than as a candidate, because parsing a name into components is not deciding that two people are the same person.
• Schema matching — deciding that two columns mean the same thing. It is the sibling problem and it is not this one.
• Data-quality expectation suites. Asserting that a column is never null is a different job from asserting that two rows are one person.

Sources and criteria#

Every fact in the profiles came from one of:

• PyPI JSON API — version, license, Requires-Python, dependency list, release dates and, critically, the wheel platform tags, which decide whether a reader needs a C compiler. 66 packages, fetched 2026-09-07.
• pypistats — installs per month, with the mirror share separated. One of this survey’s headline numbers would be dismissible as mirror traffic if it were not split, so it is.
• GitHub API — stars, forks, last push, archive status, license, and the four cells of the stall test: open pull requests, pull requests merged since 2026-03-01, open issues, and recent commit activity. 48 repositories.
• Project documentation and specifications, read rather than summarized from memory, and quoted where the exact wording is what makes the point.

The stall test is applied as four cells, not as a ratio, following this corpus’s standing rule: patches arriving with none merged is a stall; nothing arriving and nothing merged is a project that is either finished or forgotten, and the last commit tells you which. Two candidates here land in each cell, and reading them the same way would misdescribe both.

What S1 does not do#

No code, no benchmark numbers, no verdict on accuracy. S1 establishes what exists, what state it is in, and which of the four capabilities each candidate claims. Whether the claims survive contact with a labeled dataset is S2’s job, and the plan for measuring that is written before S2 begins, in ../S2-comprehensive/measurement-plan.md.

Blocking and the supporting tier#

The step that sets the ceiling, and the parsers that decide what is being compared.

Blocking gets one file because the measurements put it at the center of the category. On FEBRL3, loosening two blocking keys — same data, same comparisons, same classifier — moved recordlinkage’s exact entity recovery from 0.894 to 0.970. That single change was worth more than the entire gap between the best and second-best matcher. Nothing else in this survey has that leverage.

Why blocking is the ceiling#

Five thousand records make 12,497,500 pairs. A hundred thousand make roughly five billion. Comparing all of them is not an option, so every working system decides in advance which pairs are worth scoring.

That decision is irreversible in one direction. A true pair the blocker never proposes cannot be recovered by any model downstream. Blocking recall is therefore a hard ceiling on the whole pipeline’s recall, and it is measurable before a single comparison runs — which makes it the cheapest diagnostic in the category and the one most often skipped.

Measured on FEBRL3, three blocking keys unioned:

A 12.7% increase in pairs bought 2.6 points of ceiling. What the tools did with that extra ceiling is the finding, and it differs by tool — see ../S2-comprehensive/recommendation.md.

datasketch#

MinHash, MinHash LSH, LSH Forest, LSH Ensemble, HyperLogLog and weighted MinHash. It is the standard answer for blocking by approximate Jaccard similarity, it is by a distance the most-installed library in this file, it shipped a 2.0 major release two months ago, and it is healthy on every cell of the stall test.

It is not an entity resolver and does not claim to be. It answers “which records share enough n-grams to be worth comparing”, which is step one of four. Pairing it with a scorer and a clustering step is a legitimate architecture, and it is what several of the end-to-end tools do internally.

BlockingPy#

The newest entrant in this survey, from a Polish statistical research group: blocking via approximate nearest neighbors over several ANN backends, presented as a drop-in blocking stage for any downstream matcher. The design is right — it treats blocking as a first-class problem rather than a preamble.

It already excludes CPython 3.13, which shipped a year before this survey. A new package capped below the current interpreter is a maintenance signal worth noticing early rather than after adoption.

The parsing tier, and why it belongs in this survey#

These do not resolve entities. They decide what the comparison sees, which is upstream of every decision the rest of the category makes.

Two readings matter here.

Parsing changes the answer more than the matcher does, for addresses and names. “8236 SE 24th St” and “8236 Southeast 24th Street” have a Jaro-Winkler similarity that punishes them for a convention. Parsed into components and compared field by field, they agree exactly. No amount of threshold tuning recovers what normalization would have given for free, and this is the most common unforced error in the category.

The four-cell stall test separates these cleanly, and the answers are not what stars suggest. jellyfish shows zero inbound patches, zero merged and zero open issues with a push in July 2026 — the finished cell, the healthiest state a small library can be in. libpostal shows 18 patches waiting against 281 open issues and nothing merged — the stalled cell, on the library that does the hardest job in the list and has no replacement. Same superficial signal, opposite meanings.

What to take from this file#

Spend the first hour on blocking and parsing, not on choosing a matcher. The measurements say the ceiling is set before the model runs, and that normalization decides what the model can see. Every candidate in this survey scores better on well-blocked, well-parsed input than the best of them scores on raw input.

The database you already have#

A scored candidate, not a control. Most entity resolution that ships is written in SQL, and any library that cannot beat it has not earned its dependency.

Every relational engine in common use can express the whole pipeline. Blocking is a join condition. Scoring is an expression. Deciding is a WHERE. Clustering is a recursive CTE or a call to networkx. Nothing here requires a library, and the question a survey has to answer is what the libraries add.

What each engine gives you#

DuckDB ships jaro_similarity, jaro_winkler_similarity, levenshtein, damerau_levenshtein, jaccard, hamming, editdist3 and strip_accents as built-in scalar functions — verified against 1.5.5. There is no phonetic function: no soundex, no metaphone. It reads CSV and Parquet directly, runs in-process with no server, and is the default backend Splink itself generates SQL for. 60,187,499 installs a month, which is sixty-four times the whole dedicated tier combined.

PostgreSQL reaches the same place through two extensions that ship with the standard distribution: pg_trgm for trigram similarity with GIN/GiST index support, and fuzzystrmatch for levenshtein, soundex, metaphone and daitch_mokotoff_soundex. The index support is the distinguishing feature — a trigram similarity join in Postgres can use an index, which is a different scaling story from a full comparison.

SQLite has editdist3 in the spellfix1 extension and full-text search via FTS5, which is enough to block; fuzzymatcher is built on exactly that.

Measured, on the same benchmark as everything else#

FEBRL3: 5,000 records, 2,000 true entities, 6,538 true pairs, 12,497,500 possible pairs. DuckDB 1.5.5, aarch64, CPython 3.12.3. Full method in ../S2-comprehensive/measurement-plan.md; raw output in results/probe_duckdb_febrl3_prefix.json.

B1 — exact match on a normalized key#

Lowercase, trim, strip punctuation from the date, group on given name + surname + date of birth. Five lines of SQL, 0.005 seconds.

A precision of 1.000 is the number that gets reported, and it is the number that misleads. Nothing wrong was merged — and 84% more entities exist in the output than in reality, because everything that was not spelled identically stayed apart. Anyone validating a deduplication by sampling the merges will find no errors and will have learned nothing about the half that never merged.

B2 — blocked similarity join with a hand-picked threshold#

Block on date of birth, or postcode plus surname prefix, or surname plus given-name initial. Score with a hand-weighted sum of four Jaro-Winkler comparisons. Threshold. Connected components. About thirty lines, 0.34 seconds.

Blocking generated 8,818 candidate pairs from 12,497,500 — a reduction ratio of 0.9993 — while retaining 96.1% of true pairs. That last number is the ceiling: 3.9% of the true pairs were never proposed, and no model applied afterwards can recover them.

What this establishes#

An afternoon of SQL gets to pairwise F1 0.82 and 73% of entities exactly right. That is the bar. It is not a strawman and it is not close to useless, and a reader deciding whether to take on a dependency deserves to see it before the library results rather than after.

Precision is not the number to optimize, and the tables above show why. Every step up the threshold buys precision and loses entities: from 0.80 to 0.92, precision goes 0.954 → 0.999 and the entity count goes 2,511 → 3,031, further from the true 2,000. The conventional instinct — tighten the threshold until the false merges stop — makes the output monotonically worse at the job.

The gap between the pairwise score and the entity score is the whole subject. At threshold 0.80 the pairwise F1 is 0.815 and the exact entity recovery is 0.731. Those describe the same run. The first is what gets published; the second is what a caller writes into a table.

Where the baseline runs out#

It has no way to learn that a surname is worth more than a suburb, or that “Smith” is worth less than “Ntsekhe”. The weights above were chosen by hand and are wrong in a way nobody can see: term-frequency adjustment is the single largest thing the probabilistic engines add, and it cannot be hand-written because it depends on the data’s own distribution.

It also has no answer for the case where the threshold should differ per field, and no principled way to choose the threshold at all. The number 0.88 in the query above is a guess that survived a histogram.

dedupe#

The famous one. Active learning, a decade of citations, and a maintenance record that has stopped.

What it is#

dedupe is the library most people find first. It is the top result for “python deduplication”, it is what a decade of tutorials and conference talks reference, and at 4,510 stars it is the most-starred dedicated entity-resolution library in Python by a factor of nearly two.

Its distinguishing idea is active learning. Rather than asking the reader to configure a model, dedupe asks them to label pairs — but it chooses which pairs, picking the ones where its current model is least certain. In practice this is a console prompt showing two records and asking “same or different?”, perhaps a few hundred times, after which it has both a trained classifier and a set of learned blocking predicates. For someone who knows their data but not the statistics of record linkage, this is a good interaction design, and it is why the library earned its reputation.

It also produces clusters, with a confidence score per cluster, rather than stopping at pairs — which puts it in the small group here that finishes the job.

The maintenance picture#

The four cells of the stall test:

Fourteen people have written patches nobody has merged. The last push to the repository was 2025-07-29, over a year ago; the last release was 2024-08-15, over two years ago; 78 issues are open. This is the stalled cell, not the finished one — the difference being that people are still sending work.

The context is not a mystery and does not reflect badly on anyone. Dedupe.io, the hosted service built on the library, shut down on 2023-01-31, and its team returned their attention to consulting at DataMade. The library was always the open-source half of a commercial product, and the commercial half ended.

What follows for a reader is narrow and practical: 4,510 stars are a record of how good this library was, and the release date is the fact about whether to adopt it now.

The dependency chain is the problem#

dedupe does not stand alone. It requires a set of small compiled packages maintained by the same people:

affinegap publishes no wheel for any interpreter after CPython 3.10, on any platform. Since it is a hard dependency, installing dedupe on Python 3.11 or later requires compiling C extensions from source — which means a working toolchain and matching headers on the installing machine. That is a step that succeeds silently on a developer laptop and fails on a slim container image, and nothing on the project’s page warns of it.

Separately, dedupe 3.0.3 ships wheels for macOS, manylinux i686 and manylinux x86_64, and none for aarch64 Linux. On ARM servers and ARM containers — Graviton, Ampere, the default architecture of a container built on Apple Silicon — dedupe itself compiles from source too.

The pure-Python members of the chain are a different kind of signal. categorical-distance and highered were last released in 2016 and are still, ten years later, load-bearing imports in the most-starred library in this category.

What it is still good for#

The active-learning interaction remains the best in the category, and the algorithm is sound and well documented. A team that has already deployed dedupe, pinned its environment and is getting good results has no urgent reason to move: the code has not become wrong, and MIT licensing means a fork is available to anyone who needs one. 577 forks and fourteen unmerged patches are, viewed from a different angle, a substantial amount of already-written maintenance work sitting in a queue.

The recommendation problem is for the reader starting today, who will meet the compiler requirement in the first ten minutes and the unmerged-patch queue in the first month.

Managed services#

Named, bounded, and not ranked. The cloud vendors sell this as a product, and a reader choosing a library should know the option exists before they commit to running one.

This survey compares tools you run. Managed entity resolution is a different purchase and belongs in the 3.xxx series, so this file establishes what exists and where the boundary falls, and stops there. No vendor is scored, and no vendor’s commercial position is treated as evidence about quality.

What is on offer#

AWS Entity Resolution. Matching over data in S3 and Glue, with rule-based, ML-based and provider-service workflows. Priced per record processed. Its distinguishing property is that it stays inside an existing AWS data estate — the input is already where it runs.

Google Cloud Entity Reconciliation. Part of Enterprise Knowledge Graph, resolving records against Google’s own knowledge graph as well as against each other. Its distinguishing property is the external authority: it can tell you that two records are the same known organization, which no library here can do.

Senzing. Covered in its own file, because its SDK is on PyPI under an OSI license and that makes it reachable in a way the others are not.

Tilores, Reltio, Quantexa, Amperity, Tamr, Syniti, WinPure, Data Ladder. Commercial master-data and entity-resolution platforms, spanning API-first services through full MDM suites. Several are named in the community’s index of this category alongside the open-source libraries, which is how a reader ends up comparing a Python package with a procurement process.

The boundary that actually matters#

It is not price, and it is not accuracy. It is where the decision lives.

A library puts the matching logic in your repository. It is versioned with your code, reviewable in a pull request, reproducible from a lock file, and explicable to whoever asks why two records merged — because the reason is a model you configured and can print.

A managed service puts the matching logic behind an API. What you get back is an answer; what you keep is a bill and a vendor relationship. For many organizations that is the correct trade, and the reason is rarely technical: it is that nobody on the team wants to own a matching model for the next five years.

Three questions decide it, and all three are answerable before any evaluation:

1. Does the data leave? Records that cannot cross a boundary — health, criminal justice, anything under a data-residency rule — remove most of this column immediately. This is the most common disqualifier and it arrives first.
2. Must the decision be explicable to an outsider? Not “auditable” in the marketing sense, but: can you show a regulator, a journalist or a court why these two rows became one person. Splink’s match weight waterfall answers that; a scored API response answers it only as far as the vendor documents it.
3. Is the volume steady or spiky? Per-record pricing and a one-off migration of a legacy catalog are a bad fit for each other. The libraries’ cost is a person’s time, which is expensive and fixed; the services’ cost is per record, which is cheap until the backfill.

What this survey will not do#

It will not tell you that one vendor is stronger than another, and it will not read a funding round as a quality signal. What a company has raised answers “will this survive”, never “is this good”, and the difference is the whole reason the 1.xxx and 3.xxx series are separate.

Where a vendor fact bears on a reader’s risk it appears in that vendor’s own file as a durability signal, with its source. Senzing’s 500-record trial ceiling is the example: that is not a judgment about Senzing, it is a fact about what evaluating it costs, and a reader needs it before they start rather than after.

OpenRefine#

Not a library, and the correct answer for more readers than any library here.

Why it is in a survey of libraries#

Because excluding it would be excluding the answer. This category’s most common real situation is one person with a spreadsheet somebody sent them, who needs the same organization to stop appearing under four spellings, and who is not going to write a Python program. For that person OpenRefine is not a compromise — it is better than every library here, because the work is inspection and judgment, and a library cannot offer either.

It is also, by a distance, the healthiest project in this survey: 11,986 stars, 168 pull requests merged in six months, a release six months ago, a push yesterday. Nothing in the dedicated Python tier is maintained at that level.

What its clustering actually does#

Two families, both operating on a single column:

Key collision — compute a key per value, group values sharing a key. The keying functions are Fingerprint, N-Gram Fingerprint, and Phonetic Fingerprint, the last with Metaphone3, Cologne Phonetics, Daitch-Mokotoff and Beider-Morse variants. These are fast, deterministic and run over a whole column at once.

Nearest neighbor — Levenshtein distance and PPM. Slower, and catches variation the key functions miss.

Each cluster is presented with a suggested replacement value — the first value in the cluster by default — which the user edits or accepts. The merge is a human decision, one cluster at a time. That is the design, and it is the source of both its strength and its ceiling.

The ceiling, stated plainly#

One column. OpenRefine clusters cell values, not records. It has no notion that a row’s date of birth is evidence about whether two names are the same person. Everything this survey means by record linkage — weighting fields by reliability, combining agreement across several columns into one decision — is outside what its clustering does.

The workaround is to concatenate fields into one column and cluster that, which reintroduces the exact problem: a shared surname and a differing birth date become one string, and the string metric cannot tell which half disagreed.

It does not link two tables. OpenRefine’s reconciliation feature matches column values against an external service such as Wikidata, which is a different operation: it resolves values against a known authority, not two unknown datasets against each other.

It is manual by design. For 500 distinct company names that is a feature. For 5,000 records with 2,000 entities, the FEBRL3 workload every library here was measured on, it is a week of clicking.

The comparison#

The last row is why it wins so often. Somebody who has never done this before gets a usable answer from OpenRefine in an afternoon, and the same person spends two days configuring comparison logic in a library before seeing anything.

The row above it is why it eventually loses. OpenRefine records an operation history that can be exported and replayed, so it is not strictly one-shot — but a pipeline that must run every week against new data is a program, and at that point the work moves.

Where it sits#

The right first move for a one-off cleanup, for exploring what the messiness in a dataset actually looks like, and for any situation where a person’s judgment about each merge is worth more than a threshold. The wrong tool for anything recurring, anything multi-field, and anything at a scale where nobody will look at every cluster.

Its most under-appreciated use in this category is as a diagnostic before choosing a library: half an hour clustering the key columns in OpenRefine tells you what kinds of variation your data actually contains, which is the thing that should decide the comparison configuration in whatever you build next.

pyJedAI#

The research toolkit: every blocking and clustering algorithm in the literature, and a six-gigabyte virtual environment.

What it is#

pyJedAI is the Python successor to JedAI, a decade of entity-resolution research from the University of Athens. Where every other candidate here implements one approach, pyJedAI implements the field’s catalog: standard blocking, block cleaning, meta-blocking and comparison cleaning; similarity joins; embedding-based blocking over FAISS; a set of clustering algorithms including connected components, ricochet SR, correlation clustering and unique mapping; and — unusually — an llm_matching module with an ollama dependency.

Its module list reads like a survey of the field, because it is one. For anyone comparing blocking strategies, reproducing a paper, or teaching the subject, nothing else in Python is close. The workflow abstraction lets whole pipelines be swapped and scored against each other, which is exactly what a researcher needs and exactly what a practitioner does not.

The install is the headline#

Measured on a clean environment, aarch64, CPython 3.12.3:

The 6 GB is torch, transformers, sentence-transformers, faiss-cpu, ray, and the NVIDIA CUDA runtime — nvidia-cublas, nvidia-cudnn-cu13, nvidia-nccl-cu13, nvidia-cusparselt-cu13 and nine more. This machine has no GPU. They arrive because torch’s default wheel declares them, and pyJedAI declares torch unconditionally rather than behind an extra.

That is not an argument that pyJedAI is bad. It is an argument about where it belongs: a 6 GB image and an eleven-minute cold install are a research environment’s problem, not a production service’s, and the library makes no attempt to offer a smaller install for the parts of it that need no neural network at all.

Maintenance, read carefully#

Zero open pull requests, one merged in six months, one open issue, last push 2026-03-22.

The four-cell reading matters here. Nothing is arriving and almost nothing is being merged — that is the finished or forgotten cell, not the stalled one, and it is the cell ipazc/mtcnn occupies for good reasons. For an academic toolkit that has published its papers and implements a settled literature, low inbound traffic is close to expected. The last push six months ago and the single open issue are consistent with a project that is maintained attentively at low volume, not one that has been abandoned.

861 installs a month is the number that says who it is for. It is the least-installed candidate in this survey’s dedicated tier by an order of magnitude, and its 101 GitHub stars against Splink’s 2,386 tell the same story.

Where it sits#

The right tool for research and for teaching, and for the specific practitioner question “would a different blocking strategy help me?” — which it can answer by running six of them and scoring each, and which nothing else here can answer at all.

The wrong tool for a service that has to be deployed, on the strength of the footprint alone, and the wrong tool for a first attempt at the problem, because its breadth is organized around the literature’s taxonomy rather than around a task.

S1 verdict#

The field, sorted by how far it goes#

Every candidate placed against the four steps. “Claimed” here; whether the claims survive a labeled dataset is S2’s question.

Four things S1 established that change what to shortlist#

1. Reputation, installs and maintenance point at three different libraries.

• Most stars: dedupe, 4,510 — 0 pull requests merged in six months, last release 2024.
• Most installs: recordlinkage, 1,151,777/month — no repository commit since 2024-02-21. Not mirror traffic: mirrors are 0.1%, and the daily series has a clean weekday shape across 181 days.
• Most maintained: Splink, 210 pull requests merged since 2026-03-01, released four days before this survey.

Any one of those signals alone recommends the wrong library. The corpus’s standing rule — read the four cells, not the headline — is doing real work in this category.

2. Seven of twenty candidates cannot be imported after pip install on current Python, and an eighth installs, imports, and then raises. The eighth is dedupe: BTrees 6.5, released 2026-08-20, removed the byValue() method that dedupe/canopy_index.py:76 calls, and dedupe pins only BTrees>=4.1.4. Anyone who installed dedupe on 2026-08-19 has a working library; anyone installing today does not.

The common cause across the failures is not abandonment on its own. It is an unbounded dependency plus a stopped maintainer: matplotlib removed get_cmap, torchtext removed legacy, BTrees removed byValue. A version floor with no ceiling is a promise to keep up, and it inverts into a guaranteed future break the day the project goes quiet.

3. Two “installs” install nothing. pip install zingg takes one second and delivers a py4j bridge to a JVM that must already hold the Zingg jar. pip install senzing-core takes one second and raises failed to load the Senzing library on the first object, because the engine is separately licensed and pip does not fetch it. Both packages carry OSI license identifiers on PyPI. Neither misrepresents itself; both are easy to mistake for an installation.

4. The category’s real boundary is clustering, and most of the field stops short of it. recordlinkage — the most-installed dedicated library here — returns a pandas MultiIndex of pairs and offers no opinion about how many entities those pairs imply. string_grouper and OpenRefine cluster, but on one column. Of the candidates that cluster records on multiple fields and are installable today, there are three: Splink, pyJedAI, and dedupe with a pinned BTrees.

What proceeds to S2#

Measured end to end against ground truth: Splink, recordlinkage, dedupe, and the SQL baseline. These are the candidates that can be driven to a partition inside one container, which is the measurement plan’s cut line, and they span the three philosophies that matter — unsupervised probabilistic, toolkit-plus-classifier, and active learning — against the thing most teams actually ship.

Carried at cited, with the reason on the line: Zingg (needs Spark and a jar), Senzing (500-record license ceiling, below the field’s smallest public benchmark), OpenRefine (a desktop application, and driving it through its HTTP API would measure something no user of it does), pyJedAI (installs, but its 6 GB footprint and research framing put it outside the comparison this survey is making).

Not carried: the five that cannot be installed. They appear in the-unreachable-tier.md so that a reader meeting them in a recommendation elsewhere knows what they will find.

The question S2 has to answer#

S1 can say who claims to cluster. It cannot say whether the extra machinery is worth its cost, and the useful form of that question is not “which has the best F1”.

It is: what does a caller actually get back, and how wrong is it? A pairwise score describes pairs. A person writing one row per entity into a table lives with a partition. Those two numbers come from the same run and they are not the same number, and S2’s job is to report both for every candidate, including the SQL baseline that most teams will ship instead.

recordlinkage (Python Record Linkage Toolkit)#

The most-installed dedicated entity-resolution library in Python, and nobody has touched it since February 2024.

The number that does not fit#

recordlinkage is installed 1.15 million times a month. That is more than Splink, which is actively maintained and shipped a release four days before this survey was written, and it is twenty-seven times dedupe, which has four times as many GitHub stars.

The obvious explanations were checked and none of them hold:

• It is not mirror traffic. The mirror share is 0.1% — 1,241 of 1,153,018. Splink’s and dedupe’s shares are 0.4% and 41% respectively; recordlinkage’s is the cleanest of the three.
• It is not a spike. Across 181 days of daily data the series is stable, running between roughly 15,000 and 47,000 installs a day.
• It is not random. The series has a clean weekday shape, dropping on every Saturday and Sunday in the window. That is the signature of continuous integration and office hours — real environments being built, repeatedly, by organizations.

So the number is what it says: over a million installs a month of a library whose repository has had no commit in more than two and a half years.

What it is#

A toolkit rather than a solution, and it is explicit about it. The design follows scikit-learn: an Index object generates candidate pairs by blocking, sorted neighborhood or full index; a Compare object builds a feature vector per pair, one column per field comparison; and a classifier — logistic regression, naive Bayes, an expectation-maximization Fellegi-Sunter implementation, or an unsupervised K-means — turns feature vectors into matches.

That structure is its enduring appeal. Somebody who already knows scikit-learn can read the whole API in twenty minutes, swap in their own classifier, and inspect every intermediate artifact. For teaching the mechanics of record linkage, and for research where the point is to compare classifiers on the same feature vectors, nothing else here is as clear.

It also ships the field’s standard benchmark datasets — FEBRL 1 through 4 and the German krebsregister cancer-registry set — inside the package, with ground truth. That is quietly one of the most valuable things it provides, and it is a large part of why the library appears in so many environments: it is the easiest way to get labeled record linkage data in Python, and it is imported by coursework, papers and evaluation harnesses that never call the classifier at all. This survey uses those datasets.

Where it stops#

recordlinkage returns a MultiIndex of matched pairs. It does not cluster.

This is the boundary the survey is built on, and recordlinkage sits exactly on it. A caller with 5,000 records and a set of accepted pairs still has to decide how many entities those pairs imply, and the library offers no opinion. networkx connected components is the usual answer, and it is the wrong answer often enough to matter: one false pair can merge two distinct entities into a single component, and transitively drag in everything attached to either.

The toolkit framing means this is not a bug — it is the documented boundary of the package. But “record linkage toolkit” reads, to someone who has not done this before, like a promise to finish the job.

Maintenance, read carefully#

Three open pull requests, none merged in six months, no push since 2024-02-21, 61 open issues, last release July 2023. The four-cell reading is stalled — patches arriving, none merged — though at three inbound patches it is a quiet stall rather than dedupe’s noisy one.

The mitigating facts are real. It is pure Python with a small, conservative dependency set — pandas, numpy, scipy, scikit-learn, jellyfish, joblib — and it installs in two seconds with no compiler and no platform lottery, which is more than several maintained candidates here manage. Nothing about it has broken.

The exposure is pandas. The package pins pandas<3, and its internals use pandas idioms that have been through several deprecation cycles. A library that is not being maintained does not get to follow pandas 3, and the pin means that on the day a reader’s environment requires pandas 3, recordlinkage’s resolver contribution becomes a constraint on everything else in the environment rather than a component that upgrades.

Where it sits#

The best explanation of how record linkage works that exists in Python, still installing cleanly, still shipping the datasets the field measures itself with, and no longer being maintained. For learning, for research, and for a pipeline already built on it and pinned, it remains a reasonable place to be. As the foundation of something new that has to run in 2030, the release date is the fact that matters.

Senzing#

An Apache-2.0 badge on PyPI, in front of an engine licensed by the record.

What it actually is#

Senzing is the most technically serious entity-resolution product in this survey, and the only one that is not primarily a library. Its engine ships a large body of pre-built matching logic — name culture handling, address normalization, identifier rules, relationship detection — that a team using an open-source library would have to build. It resolves incrementally rather than in batch, which is architecturally different from everything else here: records are loaded one at a time and the entity graph updates, so there is no re-run of the whole dataset when a record arrives.

For a class of problem — ongoing resolution of people and organizations across many feeds, where somebody will ask why two records merged and the answer has to be defensible — it is the mature answer, and open-source alternatives require real work to approach it.

The license#

This is the part a reader needs before a proof of concept, not after.

The Python packages are Apache-2.0. The engine is not, and pip does not install it. Measured: pip install senzing-core succeeds in one second and then

senzing.szerror.SzSdkError: failed to load the Senzing library

on the first object constructed. Nothing is broken — the package is a binding to a native library that arrives separately.

Two further details that PyPI does not show:

• senzing 4.1.2 has no dependencies and contains no engine code. Its own README describes it as “the abstract base class from which implementations of the Senzing Python SDK are derived”, and the repository it is published from sits in the “Senzing Garage”, which the README says holds projects that are “experimental in nature” and “not considered to be ‘production ready’”. A package named after the product, at version 4.1.2, carrying an OSI license identifier, is abstract method definitions.
• The free license covers 500 records. Senzing’s own trial page states: “You’ll need an evaluation license to try with more than 500 records.” Beyond that the model is Data Source Records, priced annually, scaling with volume.

What the 500-record ceiling means for evaluation#

It is below the smallest dataset in this field’s own standard benchmark family. FEBRL1 — the toy one, the one used in tutorials — has 1,000 records. FEBRL3 and FEBRL4, used throughout this survey, have 5,000 each.

So Senzing cannot be measured on the same footing as the other candidates here without a conversation with a vendor. That is not a criticism of the product; it is a fact about the shape of an evaluation. A team comparing Senzing against Splink is not running two scripts — it is running one script and a procurement process, and the asymmetry will shape which one gets a fair trial.

This survey therefore reports Senzing’s capabilities as cited, from documentation, and says so on every line. It is not scored against the benchmark, and the reason is recorded rather than hidden.

Repository signals, read correctly#

senzing-garage/sz-sdk-python-core has one star and merged 92 pull requests in six months. sz-sdk-python has zero stars and merged 109.

The naive reading — no stars, therefore dead — is exactly wrong, and this survey uses GitHub signals throughout. Stars measure public enthusiasm. These repositories are the engineering output of a company whose customers arrive through sales rather than through GitHub, so nobody stars them and the commit volume is a paid team’s week. Read against the four-cell stall test they are among the healthiest projects in this survey.

The inverse error is the more common one, and this category supplies it too: dedupe has 4,510 stars and has merged nothing in six months.

Where it sits#

The right answer for continuous, defensible resolution of people and organizations at organizational scale, with a support contract behind it. The wrong answer for anyone who needs to know whether it works before they can talk to a salesperson, and a trap for anyone who reads Apache-2.0 on PyPI and plans accordingly.

The similarity-join tier#

Tools that find similar strings fast, are widely used, and are not entity resolvers — including several that are recommended as though they were.

These are the packages a search for “python fuzzy matching” returns. They are useful and several are well maintained. They are grouped here because they share one property: they compare one string against another string, at scale, and stop. The record has one field as far as they are concerned, the threshold is the caller’s, and the entity does not exist.

That is not a defect. It becomes a defect when the tool is adopted for a job that needs the other three steps, which is common, because the demo looks like it worked.

string_grouper#

The most interesting member of this tier, because it does more than the others. It vectorizes strings as character n-gram TF-IDF, multiplies the sparse matrices with sparse_dot_topn keeping only the top matches above a threshold, and — with group_similar_strings — returns a group label per string.

So it clusters. On one column. That is the boundary: it will happily tell you that “Michaela Neumann” and “Michael Neuman” are one group, and it has no way to be told that their dates of birth differ, or that a shared surname matters less than a shared national insurance number. Multi-field logic has to be simulated by concatenating fields into one string, which throws away exactly the per-field reliability weighting that record linkage is about.

For deduplicating a list of company names, supplier names or free-text categories — one column, many rows, no other evidence available — it is fast, actively maintained and appropriate. sparse_dot_topn compiled from source on this machine (16.0 s), so it needs a toolchain on platforms without a wheel.

name_matching#

An ING Bank library for matching company names specifically, combining several distance metrics with a TF-IDF blocking step. Installs cleanly. Its scope is its name: it is for one field, of one type, in one domain. Within that domain it encodes real knowledge — legal-form suffixes, abbreviations — that a generic string metric does not.

fuzzymatcher#

Links two pandas dataframes using SQLite’s full-text search for blocking and a probabilistic score for ranking. It installs and imports fine on 3.12 and 3.13.

It is included here for one reason: it is by the same author as Splink, and it is his earlier, smaller attempt at the same problem. Reading the two together is the clearest available illustration of what the extra machinery in Splink is for. It is not being maintained, has never had a 0.1 release, and its own author moved on — which is about as clear a signal as this category produces.

Zero inbound patches and zero merged puts it in the finished or forgotten cell rather than the stalled one; the 2022 last push says which.

polyfuzz#

A well-designed unified API over several string-matching backends — TF-IDF, edit distance, embeddings, RapidFuzz — with a common interface and evaluation helpers. It is the tidiest piece of API design in this tier.

It also raises ImportError: cannot import name 'get_cmap' from 'matplotlib.cm' on import, because Matplotlib removed get_cmap in 3.9 and PolyFuzz declares no upper bound. Four open pull requests, none merged since 2026-03-01. Details in the-unreachable-tier.md.

csvmatch#

A command-line tool for fuzzy-joining two CSV files, from a data journalist, and shaped for that job: it is meant to be run from a terminal against two files someone has just been sent. That framing is a real gap in this category — every other candidate assumes a Python program is being written.

The exact-minor Python pin is the problem. ==3.13.* is not a floor; on 3.12 the resolver backtracks to an older release whose dependency chain no longer builds, and on 3.14 the current release will stop being installable entirely.

What this tier is for#

Use it when the record is one string — a column of names to normalize, a list of free-text values to group, a join key that came from a spreadsheet. In that situation these tools are the right size, and reaching for Splink would be over-engineering.

Stop using it the moment a second field carries evidence. Two people with the same name and different birth dates are the case that separates the tiers, and no threshold on a string similarity will ever get it right, because the information needed is not in the string being compared.

Splink#

Probabilistic linkage that compiles to SQL and runs wherever your data already is.

What it is#

Splink implements the Fellegi-Sunter model — the 1969 statistical framework that formalised record linkage — and then does something structurally unusual with it: rather than computing comparisons in Python, it generates SQL and hands it to a database engine. The model is the same whichever engine receives it; the engine decides how big the problem can get.

The supported backends are DuckDB (in-process, the default), PySpark, AWS Athena, SQLite and PostgreSQL. DuckDB arrives as a hard dependency; the rest are extras. The practical consequence is a single library that covers the laptop case and the cluster case with the same model code, which nothing else in this survey does.

Why it is unusual among open-source data tools#

It is maintained by a government department. Splink comes out of the UK Ministry of Justice’s analytical services team, which built it to link criminal justice records and then published it. Two things follow, and they pull in opposite directions.

The good half is visible in the maintenance record. 210 pull requests merged in six months against 25 open is the healthiest ratio in this category by a wide margin — the project is not one person’s evening hobby, and the deprecation of a version is announced rather than discovered. It is used in production by an organization that has to explain its methodology in public, which is why the diagnostics are unusually good: match weight waterfalls, parameter estimate charts, and a comparison viewer that shows why a pair scored what it did. Very little else here can show its working.

The other half is that the roadmap answers to a public-sector data problem. Splink’s model of the world is people, addresses and dates, because that is what a justice ministry links. Nothing prevents its use on products or companies, and people do it, but the term-frequency adjustments and the built-in comparison library are shaped around the first case.

Unsupervised training, and its caveat#

Splink estimates its own parameters with expectation-maximization, so no labeled training data is required. Its documentation states this plainly: “No training data is required, as models can be trained using an unsupervised approach.”

This is the single largest practical difference between Splink and the active-learning tier. Producing labeled pairs is the expensive part of an entity-resolution project — somebody has to sit and answer “same person?” several hundred times — and Splink removes it.

The caveat is that unsupervised does not mean unconfigured. The reader still specifies which fields to compare, how to compare each one, and which blocking rules to use, and those choices matter more to the outcome than the estimated parameters do. “No training data” and “no expertise” are different claims, and only the first is made.

Adoption, and what the number is really counting#

945,285 installs a month is the second-largest figure in this survey’s dedicated tier and the largest for any maintained project in it. The mirror share is 0.4%, so it is people and machines installing it, not a caching proxy.

Set against 2,386 GitHub stars, this is an unusual ratio: roughly 400 installs a month per star. The pattern is characteristic of a tool that gets adopted inside organizations rather than admired on social media, which is consistent with its origin.

Costs and constraints#

A SQL backend is not free of consequences. The library generates SQL, so debugging a slow linkage sometimes means reading generated SQL, and the engines are different — a model that runs comfortably on DuckDB in-process is not automatically tuned for Spark. The design intent is portability of the model, not of the performance profile.

Version 4 was a rewrite of the API. Splink 3 code does not run on Splink 4, and tutorials, blog posts and Stack Overflow answers written before the change are a trap for a newcomer. The project documents the migration, but the search-engine sediment is several years deep and does not announce its own version.

sqlglot is load-bearing. SQL generation goes through it, which means a dialect gap in a transitive dependency can surface as a backend-specific bug rather than as a Splink bug.

Where it sits#

The strongest general-purpose starting point in the category, and the only candidate here that performs all four steps under one API while running on infrastructure a team already has. Its weakest moment is the first hour: the configuration surface is large, and the model asks the reader to make decisions about field comparison before showing any result.

The unreachable tier#

Seven of twenty candidates cannot be imported after a clean pip install on current Python. Most of them are still being recommended.

This file exists because a survey of this category that listed only what the field names would be misleading. The community’s own index of entity-resolution software lists DeepMatcher, Entity Embed, py_entitymatching and csvmatch alongside Splink and dedupe, without distinguishing between them. On a clean machine in September 2026 they are not the same kind of object.

Measured on aarch64 Linux, CPython 3.12.3 and 3.13.15, one clean virtual environment per candidate, uv 0.12.5. The full matrix and every error is in harness/1-194-entity-resolution-record-linkage/results/install.json.

Cannot resolve#

py_entitymatching (Magellan, University of Wisconsin — 194 stars, 0.4.2 from 2024-02-07). Its dependency py-stringsimjoin fails to build: ImportError: cannot import name 'msvccompiler' from 'distutils'. Modern setuptools removed msvccompiler, and the package’s setup.py imports it unconditionally on every platform, including Linux where it was never relevant. Three open pull requests, none merged since 2026-03-01, 47 open issues.

entity-embed (Vinta Software — 162 stars, 0.0.6 from 2021-07-16). Requires torchtext, which stopped shipping wheels after CPython 3.11 and whose last release targets torch 1.8.1. The resolver’s own explanation is unusually clear: “we only found wheels for torchtext (v0.16.1) with the following Python ABI tags: cp38, cp39, cp310, cp311”. There is no version of this package that installs on a supported Python.

nomenklatura (OpenSanctions — 265 stars, 4.14.0 from 2026-08-23, actively maintained, 60 pull requests merged in six months). This one is not abandoned at all: it fails because rigour → normality → pyicu needs the ICU C library and pkg-config installed on the machine. Please install pkg-config on your system or set the ICU_VERSION environment variable. A healthy project with a system-library dependency is a different problem from a dead one, and the fix is apt install libicu-dev pkg-config — but it is a fact about adoption cost, and it is invisible from PyPI.

csvmatch (190 stars, 2.1.0 from 2025-02-02). Its published version declares Requires-Python: ==3.13.* — pinned to exactly one CPython minor release, not a floor. On 3.12 the resolver therefore backtracks to csvmatch 2.0.1, whose chain reaches doublemetaphone 1.1, which fails to cythonize. On 3.13 it fails differently. An exact minor pin means the package’s usable life is one Python release long.

Installs, then will not import#

polyfuzz (MaartenGr — 802 stars, 0.4.3 from 2025-05-06, 36,338 installs a month). Resolves and installs in under a second, then:

from matplotlib.cm import get_cmap
ImportError: cannot import name 'get_cmap' from 'matplotlib.cm'

get_cmap was removed from Matplotlib in 3.9. PolyFuzz declares no upper bound, so a fresh environment gets a current Matplotlib and a package that cannot be imported at all — the failure is at import polyfuzz, before any user code runs. Four open pull requests, none merged since 2026-03-01.

deepmatcher (University of Wisconsin — 623 stars, 0.1.2 from 2021-06-13). ModuleNotFoundError: No module named 'torchtext.legacy'. The legacy module was removed from torchtext in 2022. Six open pull requests, none merged, 69 open issues. This is the package a decade of “deep learning for entity matching” papers benchmark against.

senzing-core — a different case, and the important one. It installs cleanly, and then:

senzing.szerror.SzSdkError: failed to load the Senzing library

Nothing is broken. The Python package is a binding, and the engine it binds to is a separately licensed native library that pip does not install. See senzing.md.

Installs, imports, then fails at runtime#

dedupe is in its own category and is covered in dedupe.md. In summary: it installs, it imports, and it raises AttributeError: 'BTrees.IFBTree.IFBucket' object has no attribute 'byValue' the first time it blocks. BTrees 6.5, released 2026-08-20, removed the long-deprecated byValue(); dedupe pins BTrees>=4.1.4 with no ceiling. The library was working for anyone who installed it on 2026-08-19 and is not for anyone who installs it now.

How to read this#

A stalled project is not the same as a broken one, and the difference is testable. recordlinkage has had no commit since February 2024 and installs in 1.7 seconds with no compiler; rltk has not released since 2021 and imports fine. Neither has become wrong. What kills a package is not silence — it is silence plus an unbounded dependency on something that keeps moving. Three of the failures above are exactly that shape: matplotlib removed a function, torchtext removed a module, BTrees removed a method, and in each case a library with no maintainer had declared no ceiling.

The corollary for a reader choosing today: a lower bound with no upper bound is a maintenance promise. It says the project intends to keep up. When the project stops, the promise inverts and becomes a guarantee of eventual breakage, on a date nobody chooses and nobody announces.

And for a reader already in production: none of this reaches a pinned, working environment. Everything above is a fact about installing fresh. The lock file that looks like technical debt is, in this category, the thing standing between a working pipeline and the September 2026 BTrees release.

Zingg#

Active learning on Spark, AGPL-3.0, and a PyPI package that is a telephone line rather than a tool.

What it is#

Zingg is a Spark-native entity resolver built around active learning: it proposes uncertain pairs, a human labels perhaps twenty to forty of them, and it learns both the blocking model and the matching model from those labels. Conceptually it is dedupe’s approach at cluster scale, and it is the only candidate in this survey designed from the start for datasets that do not fit on one machine.

It also covers a step most of the field ignores: it produces a linked, mastered output with a cluster identifier per record, and it is explicit about incremental runs — new data resolved against an existing set of entities rather than a full re-run.

The Python package is a bridge, not the product#

pip install zingg completes in one second and pulls in py4j and nothing else.

That is the whole package. Zingg is a Java/Scala application that runs on Spark; the PyPI distribution is a thin py4j wrapper that talks to a JVM which must already have the Zingg jar. Installing the Python package does not install Zingg, in the same way that installing senzing-core does not install Senzing — and in both cases the install succeeds, which is what makes it a trap.

The 1,836 installs a month should be read against that. It is not a measure of Zingg’s adoption; it is a measure of how many people reach Zingg through Python, which for a Spark tool is a minority path. The 1,243 stars are the better adoption signal, and 82 open issues against 56 merged pull requests in six months is a real, working project.

AGPL-3.0 is the fact to decide on first#

Zingg is the only candidate in this survey under a copyleft license with a network clause. Every other library here is MIT, BSD or Apache.

What that means in practice depends entirely on deployment shape, and the survey will not pretend otherwise:

• Running Zingg as an internal batch job over your own data, publishing nothing: the obligation is not triggered by internal use.
• Building a product where users interact with something Zingg is part of, over a network: the AGPL’s section 13 is written for that case, and the conventional reading is that source must be offered to those users.

The company sells Zingg Enterprise, which is how the AGPL is being used here — as the standard open-core boundary. That is a legitimate business model and not a criticism. It is simply a decision that has to be made by someone with authority to make it, before any technical evaluation, and it is the one candidate here where that is true.

Costs#

A JVM and a Spark distribution. For a team already on Spark this is nothing; for a team that is not, it is the entire adoption cost, and it dwarfs the library itself.

It is not measurable in this survey’s harness, and that is why. Reaching Zingg requires a second runtime and a downloaded engine, which is above the measurement plan’s cut line. Its claims here are cited and marked as such.

The scale it is built for is the scale at which its competitors are also fine. Splink runs on PySpark too, with the same model it runs on DuckDB, and under MIT. That comparison is the one a Spark-based team should actually make, and it is not a comparison Zingg’s own documentation frames.

Where it sits#

The right shortlist entry for a team that is already on Spark, wants active learning rather than unsupervised estimation, and can either satisfy the AGPL or buy the commercial license. Otherwise the Python package’s one-second install is misleading about what adoption costs.

S2 approach#

What was run#

Four candidates driven end to end against ground truth, in their own virtual environments, on two datasets, under two blocking configurations. Everything else in the survey is cited and marked as such on the line.

The data#

FEBRL — the Freely Extensible Biomedical Record Linkage generator’s standard datasets, the benchmark this field has measured itself with for two decades. They ship inside recordlinkage, which is one of the candidates, so export_data.py freezes them to CSV with SHA-256 digests before any probe runs. No candidate supplies the data it is tested on.

FEBRL1 is the easy case: every entity is exactly two records. FEBRL3 is the one that matters, because its clusters run from one record to six — 835 singletons, 368 pairs, and 168 entities with six records each. That distribution is what makes clustering a real step rather than a formality, and it is why the two datasets rank the candidates differently.

Two numbers per run, never one#

Every result is scored twice by one shared scorer (scoring.py, imported into all four environments, pure stdlib because they share no dependencies):

Pairwise — precision, recall and F1 over the set of record pairs asserted to be the same entity. This is what the field reports.

Cluster — the number of entities returned, how many true entities came back as exactly the right set of records, and the Adjusted Rand Index over the partition. This is what a caller writing one row per entity actually lives with.

They are not derivable from each other, and the gap between them is this survey’s subject. A single false pair costs one unit of pairwise precision and welds two true entities into one, which costs two entities. Reporting only the first flatters every tool that stops before clustering.

Fairness rules, and the one that was broken and fixed#

One set of blocking rules for all candidates. The first pass did not do this. The SQL baseline was given prefix-based blocking keys while the libraries were given whole-field keys, so the libraries lost recall to the harness rather than to their own design. Both variants were then run for all candidates and both are committed. The difference turned out to be larger than the difference between the candidates, which is why it became a finding rather than a correction.

One set of comparison fields for all candidates. Given name, surname, date of birth, suburb, postcode, address line 1. Six fields, chosen because every candidate can express comparisons on all six.

Unsupervised where unsupervised is possible. recordlinkage is scored on its ECM classifier, not its supervised ones, because Splink and the SQL baseline get no labels either. Giving one candidate training pairs and not the others would measure the labels.

dedupe’s active learning is answered by an oracle, and that is declared. dedupe is built around a human answering its uncertain_pairs() queries. A human cannot sit inside a harness, so the probe answers from ground truth — 40 labels on FEBRL1, of which 28 were matches. This is dedupe’s own interaction with a perfect labeler, so the result is a ceiling on what the active-learning loop achieves, not a prediction of what a tired analyst gets on a Friday afternoon.

Thresholds are swept, not chosen. Every candidate is reported at four operating points. Picking one threshold per tool and comparing would be comparing the author’s taste.

What is deliberately not measured#

Throughput at scale. 5,000 records is not a scale test and the survey makes no scale claims from it. What can be said about scale is architectural — Splink pushes work into a SQL engine, Zingg into Spark — and it is said as architecture, at cited.

Accuracy on data that looks like yours. FEBRL is synthetic Australian personal data with generated corruptions. The ranking it produces is informative; the absolute numbers are not a forecast for a catalog of museum objects or a list of company names. Where the survey generalizes, it generalizes about mechanisms — that blocking sets a ceiling, that a fixed threshold cannot absorb extra candidates — because those follow from how the tools work rather than from this dataset.

Anything needing a second machine or an external service. The measurement plan’s cut line, held to.

Conditions attached to every number#

aarch64 Linux (WSL2, kernel 6.6.87.2), CPython 3.12.3 and 3.13.15, uv 0.12.5, DuckDB 1.5.5, measured 2026-09-07. Package versions in requirements.txt; raw JSON for every claim in results/. Architecture is reported rather than assumed away: two candidates publish no aarch64 wheel and compile from source here while installing from a wheel on x86_64, and where that changes the answer the survey gives both.

Blocking: the step that decides the outcome#

Changing the blocking rules moved the result more than changing the library did.

This is S2’s central measurement and it reorders the category’s priorities. The conventional shape of an entity-resolution decision is “which matcher?” The measurements say the matcher is the second question.

The arithmetic that forces the step#

Comparing every pair is quadratic and stops being possible almost immediately.

Every working system therefore proposes a subset — candidate pairs — and scores only those. Two numbers describe how well:

Reduction ratio: 1 − (candidates / possible). How much work was avoided. Blocking recall: the fraction of true pairs that made it into the candidate set.

The second is the one that matters, and it behaves differently from every other number in the pipeline: it is a hard ceiling. A true pair the blocker never proposes cannot be recovered by any model, threshold or clustering algorithm downstream. Recall lost at blocking is lost permanently, and it is lost before anything visible has happened.

Measured: two blocking configurations, FEBRL3#

Identical data, identical comparison functions, identical classifiers. Only the blocking keys changed.

Configuration A — whole-field keys. The obvious first choice: block on date of birth; on postcode plus surname; on surname plus given name.

Configuration B — two keys loosened to a prefix. Block on date of birth; on postcode plus the first two characters of surname; on surname plus the first character of given name.

A 12.7% increase in pairs bought 2.6 points of ceiling. Both configurations discard over 99.8% of the work.

What each candidate did with the extra ceiling#

Exact entity recovery on FEBRL3 — the fraction of the 2,000 true entities returned as exactly the right set of records:

Three things follow, and the third is the one worth carrying away.

Blocking outweighs the matcher. recordlinkage moved 7.6 points on blocking alone. The gap between recordlinkage and Splink at configuration B — the best each can do — is 0.7 points. A team choosing between those two libraries is arguing about a tenth of what they would gain by spending the same hour on blocking rules.

The ceiling is not the only thing blocking changes. Recall rose 2.6 points and recordlinkage’s entity recovery rose 7.6. Extra candidate pairs do more than add recoverable true pairs; they change what the model estimates from. The ECM estimator sees a different distribution of agreement patterns and produces different m and u probabilities, and in this case better ones.

Better blocking made the baseline worse. This is the result that was not expected and it is the most useful one in the survey.

Why more candidates hurt a fixed threshold#

Blocking adds true pairs and false ones together. What happens next depends entirely on whether anything downstream can tell them apart.

A learned model re-estimates its weights over the enlarged candidate set. Agreement on a common surname is worth less when the candidate set contains more common-surname pairs, and the model discovers that from the data. It absorbs the additional false candidates by scoring them lower.

A hand-picked constant cannot. score >= 0.80 means the same thing before and after, so every additional false candidate that clears 0.80 becomes a false pair, and connected components turns each false pair into a merged entity. The baseline gained recall and lost more precision than the recall was worth: 0.740 → 0.731.

This is the concrete answer to “why not just write the SQL”. Not that the model scores better on the same candidate pairs — at configuration A the SQL baseline and the ECM classifier are 15 points apart, which is a lot, but the model is also 30 lines of configuration against 30 lines of SQL. The answer is that the model is what makes better blocking safe to adopt. The baseline is stuck at whatever blocking its threshold was tuned against, and improving one requires re-tuning the other.

The diagnostic every project should run first#

Blocking recall is measurable before any model exists, on any labeled sample, and it costs one join:

1. Take whatever ground truth you have — even a few hundred hand-checked pairs.
2. Generate candidate pairs with your proposed blocking rules.
3. Count how many of the known-true pairs appear.

If that number is 0.85, the best possible recall of the finished system is 0.85, and no amount of model tuning will move it. Finding that out in the first hour is the cheapest thing in this category. Finding it out after building the model is the most common way an entity-resolution project produces a disappointing result nobody can explain.

How the candidates express blocking#

Splink takes block_on(...) rules that become SQL join conditions, and accepts arbitrary SQL expressions — block_on("postcode", "substr(surname,1,2)") is a rule. Rules are unioned and deduplicated by the library. It also estimates its own probability_two_random_records_match from a set of blocking rules and a stated recall assumption, which makes blocking part of the model rather than a preamble to it.

recordlinkage offers Index.block on one or more columns, Index.sortedneighbourhood for a sliding window over a sorted key, and Index.full for the quadratic version. Prefix blocking requires deriving the prefix column yourself first, which is a small friction that discourages exactly the change that gained 7.6 points.

dedupe learns its blocking predicates from the labeled pairs, which is its most distinctive property: the reader never writes a blocking rule. Measured on FEBRL1 it achieved blocking recall 0.998 — the best in the survey — from 40 labels. On FEBRL3 the predicate learner is also where it ran out of memory, in coveredPairs, before any labeling happened. Learning the rules is worth a great deal and costs a great deal, and both halves showed up in the same measurement.

The SQL baseline expresses blocking as a JOIN ... ON with a disjunction, which is the most legible form of it anywhere in this survey — the rules are visible, reviewable and modifiable by anyone who reads SQL, with no library-specific vocabulary in between.

Clustering, and the pairwise illusion#

Two numbers come out of every run. The field reports one of them, and the other is what you ship.

The illusion, in one table#

Every row below is a single run on FEBRL3 — 5,000 records describing 2,000 real entities. The left half is what a paper or a project page reports. The right half is what a caller gets back.

A pairwise F1 of 0.815 sounds like a system that mostly works. The same run returns 2,511 entities where there are 2,000, and gets 27% of them wrong. Both descriptions are accurate. Only one of them is about the output.

Why the two numbers diverge#

A false pair does not cost one unit of error. It costs a merge.

If A and B are one person and C and D are another, and the matcher wrongly accepts B~C, then connected components produces one cluster of four. That is one false pair — a small dent in pairwise precision, one row among thousands — and it has destroyed two true entities, replacing them with one wrong one.

The asymmetry runs the other way too, and less violently. A missed pair splits one entity into two, which is one wrong entity rather than two. So under-merging degrades the partition gracefully and over-merging degrades it catastrophically, and the pairwise F1 weights them the same.

This is why precision matters more than recall in this category, and by more than the usual amount. Measured directly: at configuration A on FEBRL3, Splink scored pairwise precision 0.999 and recordlinkage’s ECM scored 0.965 — a gap of 3.4 points. Their exact entity recovery was 0.959 and 0.894 — a gap of 6.5 points, nearly double. The clustering step amplified the precision difference, exactly as the mechanism predicts.

The entity count is not a validation#

The most tempting check is the cheapest: count the entities and see whether the number is plausible.

Measured on FEBRL3 under configuration A, recordlinkage’s ECM classifier returned exactly 2,000 entities, where the truth is 2,000. Splink returned 2,079, overshooting by 3.95%.

Exact entity recovery: recordlinkage 0.894, Splink 0.959.

The tool that got the count perfectly right got a tenth of the entities wrong. Its over-merges and under-merges canceled in the total and did not cancel in the partition. A reader validating by count would have picked the worse result, and would have felt well-informed doing it.

The same trap runs the other way at the bottom of the table. Exact matching on a normalized key scores precision 1.000 — nothing was wrongly merged, and a reviewer sampling the merges will find no errors at all. It returns 3,682 entities where there are 2,000. Every error is in the half that never merged, which is the half nobody inspects.

What the candidates actually do to cluster#

Measured, not read from documentation.

Splink’s clustering is connected components. Its own cluster_pairwise_predictions_at_threshold produced a partition identical to naive connected components over the same accepted pairs, at all four thresholds tested — 2,041, 2,044, 2,048 and 2,055 entities, matching in every case, with identical exact-recovery and ARI figures.

That deflates a natural assumption. Splink’s advantage over the SQL baseline is not a cleverer partitioning algorithm. It is pair precision: 0.999 fed into connected components beats 0.954 fed into the same algorithm. The sophisticated part is upstream, and the clustering is the same transitive closure a caller would have written — done correctly, in the database, at whatever scale the backend supports, which is a real convenience but not a different idea.

dedupe is the exception. Its cluster() performs hierarchical clustering and admits a cluster only if the cophenetic similarity of its members clears the threshold. By construction that refuses the A–B–C chain: B may be close to both A and C, but if A and C are far apart the cluster does not survive. On FEBRL1 at threshold 0.3 the difference was visible and small — dedupe’s hierarchical partition returned 502 entities against connected components’ 501 on the same scores, recovering 99.6% exactly against 99.2%.

On easy data the two agree. The mechanism predicts they diverge as chains get longer, which is FEBRL3’s regime — clusters up to six records — and FEBRL3 is where dedupe ran out of memory. That comparison is the most interesting one this survey could not complete, and it is recorded as a gap rather than guessed at.

recordlinkage does not cluster at all. It returns a pandas MultiIndex of matched pairs. Every partition reported for it in this survey was produced by the harness taking connected components, which is what a caller does. The library is a toolkit and this is its documented boundary — but “record linkage toolkit” reads, to somebody who has not done this before, like a promise to finish, and the step it leaves out is the one where a single false pair does double damage.

The SQL baseline uses connected components too, either as a recursive CTE or by handing the accepted pairs to networkx. Same algorithm, same amplification, worse input.

The practical consequence#

Report entity-level metrics, or you are not measuring the thing you are shipping. Pairwise precision, recall and F1 are the field’s convention and they describe an intermediate artifact. The number of entities returned, the fraction recovered exactly, and an ARI over the partition take ten lines to compute once ground truth exists, and they are what the downstream table will contain.

Tune for precision, then widen the blocking. Since clustering amplifies false pairs and merely accumulates missed ones, the cheap route to a better partition is a high threshold plus better blocking — not a lower threshold. The measurements show both halves: every step down the SQL baseline’s threshold from 0.92 to 0.80 improved its partition, and every improvement in blocking helped the models that could re-weight and hurt the one that could not.

And do not trust a count. It is the one validation that costs nothing and the one that told the clearest lie in this survey.

dedupe — architecture and measurements#

The idea that made it famous#

dedupe does not ask the reader to configure a model. It asks them to answer questions, and it chooses the questions.

prepare_training builds a set of candidate blocking predicates and an active learner. uncertain_pairs() returns the pairs the current model is least sure about — the ones near the decision boundary, where a label carries the most information. mark_pairs() records the answer. Repeat a few dozen times, then train(), and the library has learned both a matching model and a set of blocking predicates.

Learning the blocking predicates is the part nothing else here does. Every other candidate requires the reader to write blocking rules, and blocking.md shows that those rules move the result more than the matcher does. dedupe removes that decision, and on FEBRL1 it made it better than the author did: blocking recall 0.998, against 0.980 for the hand-written rules used by the other candidates on the same data.

The full capability set#

dedupe is the only measured candidate that performs all five steps.

Blocking — learned predicates plus TF-IDF canopies, indexed. Scoring — a regularized logistic regression over field-comparison distances. Deciding — a threshold on the pairwise score. Clustering — hierarchical, gated on cophenetic similarity. Not connected components: a cluster survives only if its members are mutually close, so the A–B–C chain is refused by construction rather than by tuning. Canonicalization — dedupe.canonicalize() returns one representative record per cluster.

The canonicalization step is worth a concrete example, because it is the capability readers most often assume exists everywhere and which exists almost nowhere. From the measured run on FEBRL1:

The canonical record recovered a date of birth that one input lacked. That is the point of the step: the surviving row is not a copy of one input, it is assembled from the cluster, field by field. A caller who has to write one row back into a system gets something usable instead of a decision to make.

Measured — FEBRL1#

1,000 records, 500 true entities. 40 labels, answered from ground truth (a perfect oracle, so read these as a ceiling on the human loop). Training 9.38 s, scoring 0.43 s, peak resident memory 524 MiB.

This is the best result in the survey on this dataset — better than Splink’s 0.989 and recordlinkage’s 0.989 — and it comes from 40 labels and learned blocking. Where the active learning loop can be run, it works.

At threshold 0.3 the hierarchical clustering also beat connected components on the identical scores: 502 entities and 99.6% exact against 501 and 99.2%. Small, and in the direction the mechanism predicts.

Measured — FEBRL3: it did not finish#

5,000 records. Library defaults for prepare_training (sample_size=1500, blocked_proportion=0.9), 6 GiB address-space cap:

File ".../dedupe/training.py", line 177, in coveredPairs
    pairs = frozenset(
MemoryError

The failure is inside prepare_training → DedupeDisagreementLearner → DedupeBlockLearner → coveredPairs, before any labeling happens. An earlier uncapped run on the same input reached 7.9 GiB resident and was killed to protect the machine.

For scale: Splink completed the same dataset — training, prediction and clustering — in 2.24 seconds inside a 191 MiB environment.

This is one configuration on one dataset, and dedupe’s own documentation says partition() is for “small to moderately sized datasets” and directs larger data through pairs()/score() manually. But the memory went during predicate learning, which is not the step that documentation is about, and 5,000 records is not a large dataset by any reading. The measurement is reported with its exact conditions and a reader can reproduce it.

The two blockers on adopting it today#

A fresh install does not work. dedupe 3.0.3 pins BTrees>=4.1.4 with no ceiling. BTrees 6.5, released 2026-08-20, removed the long-deprecated byValue() from all Bucket and BTree classes except the II and LL families — the changelog gives the cause as a C-level crash, issue 226. dedupe calls results.byValue(...) at canopy_index.py:76, on an IFBucket, which is in the removed set. The result:

AttributeError: 'BTrees.IFBTree.IFBucket' object has no attribute 'byValue'

raised the first time it blocks. Bisected: 5.2, 6.0, 6.1, 6.2, 6.3 and 6.4 all have byValue; only 6.5 does not. BTrees==6.4 makes the same code run, and every measurement above used that pin.

It needs a C toolchain. affinegap publishes no wheel past CPython 3.10 on any platform, and dedupe itself publishes no aarch64 Linux wheel at any version. On this machine the install compiled affinegap, dedupe-levenshtein-search and dedupe from source (and pylbfgs as well on 3.13). On x86_64 with Python 3.12 or 3.13, dedupe’s own wheel exists but affinegap still compiles.

What this means#

The algorithm is good, the interaction design is the best in the category, and it is the only candidate that finishes the job all the way to a canonical record. It also has 14 patches waiting with none merged in six months, no push since 2025-07-29, and a dependency chain whose oldest member was last released in 2016.

A team already running dedupe should pin BTrees<6.5 and change nothing else. A team starting today is adopting a library whose next breakage has the same shape as the last one and no one to fix it.

Feature comparison#

Measured where the cell says measured. Cited where it says cited, with the reason a measurement was not possible in measurement-plan.md.

The four steps#

Two cells carry most of the decision. recordlinkage does not cluster, which is the step where a false pair does double damage. dedupe is the only one that canonicalizes, and the only one whose clustering is not the transitive closure.

How the decision gets made#

The labels required column is the practical axis. Producing labeled pairs is the expensive input in an entity-resolution project, and it is the reason the unsupervised tier is the default recommendation despite dedupe scoring higher where it runs.

Accuracy, both datasets, best operating point#

exact is the fraction of true entities returned as exactly the right set of records. FEBRL1 is 1,000 records with 500 two-record entities; FEBRL3 is 5,000 records with 2,000 entities of size 1 to 6.

The datasets rank the candidates differently, and the reason is the reason FEBRL3 is the one to read. On the easy set dedupe wins and Splink and recordlinkage tie exactly. On the set with long clusters, dedupe does not finish and Splink pulls ahead.

Adoption cost#

Cold install on CPython 3.12, aarch64. pyJedAI’s 5,976 MiB includes the full NVIDIA CUDA runtime on a machine with no GPU.

Licensing#

Two of these are a decision before an evaluation rather than after one.

Maintenance — the four cells#

Open pull requests against pull requests merged since 2026-03-01, fetched 2026-09-07.

Stars and health are uncorrelated here in both directions. The most-starred library in the category has merged nothing in six months; the two Senzing repositories have one star between them and merged 201.

Installs per month, mirrors excluded#

The primitives dwarf the resolvers by two orders of magnitude, which is itself the category’s situation: far more people are comparing strings than resolving entities, and some of them think those are the same job.

Measurement plan#

Written before S2, per Step 3.5. Levels are listed cheapest first. Each states what it settles, what it costs, the rung on the evidence ladder it can reach, and how many of the surveyed candidates it can actually touch.

The cut line is the default one: everything that runs in one container, with no external service and no second machine. Three levels sit above it and are deferred with a reason.

The question the levels exist to settle#

S1 divides the field on one claim: a tool that scores a pair of records has not resolved an entity. Between those two things sit three steps — blocking, deciding, and clustering — and the survey’s assertion is that most of the field stops after the first or second, while the phrase “entity resolution” promises all three.

That is a capability claim, so the load-bearing level is L2, not the accuracy run. An accuracy number for a tool that never clusters is measuring a different task from an accuracy number for one that does, and reporting them in one column would be the error this plan exists to avoid.

L0 — Registry facts#

registry.py, registry2.py, downloads.py, mirrors.py → results/registry.json, registry2.json, mirrors.json.

The mirror split is not decoration. One candidate’s install volume is the survey’s most counterintuitive number, and “it is mirror traffic” is the first thing a reader should suspect. It is measured and it is not.

L1 — The install matrix#

install_matrix.py → results/install.json.

Recorded per run: resolution outcome, install seconds, which distributions had to be compiled from source, the imported version, and the full freeze. The source-build list is the point. A package with no wheel for the reader’s interpreter and architecture needs a C toolchain, and that requirement is invisible on the project’s own page.

Architecture is part of the result, not a footnote. This machine is aarch64. A missing aarch64 wheel is a real constraint for anyone on Graviton, Ampere or an Apple Silicon container, and it is not a constraint on x86_64. Where the two diverge, the survey reports the wheel tags for both rather than generalizing from one run.

L2 — The capability probe: block, decide, cluster#

The five capabilities are recorded as observed behavior — what the returned object actually contains — not as documentation claims. A library that documents clustering and returns pairs is the case this level exists to catch.

L3 — Accuracy against ground truth#

Data. FEBRL, the standard public benchmark for this field, distributed inside recordlinkage:

• FEBRL3 — 5,000 records, 2,000 true entities, cluster sizes 1 to 6, 6,538 true pairs. The deduplication case, and the one where clustering matters.
• FEBRL4 — 5,000 × 5,000, 5,000 true links, one-to-one. The linkage case.

Both carry ground truth, so precision, recall and F1 are computed rather than asserted. Cluster-level agreement is scored separately from pairwise.

The baseline is a scored candidate, not a control. Exact match on a normalized key, and a pg_trgm-style similarity join in SQL, are what most teams actually ship. They go in the same table as the probabilistic engines.

L4 — Blocking economics#

Blocking recall is the headline here: a blocking rule that discards a true pair has set a ceiling on the whole pipeline that no downstream model can lift, and it is the failure that does not show up in a match-quality score.

Above the cut line — deferred, with reasons#

These are filed rather than dropped. Each names what would have to change.

Spark backends (Zingg, Splink-on-PySpark, spark-matcher). Needs a JVM, a Spark distribution and a downloaded engine jar; Zingg’s actual product is not the PyPI package. Claims about their scale behavior stay at cited and say so on the line.

Postgres and Athena backends for Splink. A second service and, for Athena, an AWS account. The DuckDB backend is in-process and is measured; the survey does not generalize its numbers to the others, because the whole design point of the SQL-backend architecture is that the engine differs.

Senzing. Unreachable, and the reason is the finding: the free license that ships with the SDK covers 500 records, which is below the smallest dataset in the field’s own standard benchmark family (FEBRL1, 1,000 records). Evaluating it on the same footing as everything else here requires contacting a vendor. Its claims are cited, and the 500-record ceiling is reported as a fact about adoption cost.

OpenRefine. A desktop application driven by a browser, not a library. It belongs in the survey — it is the right answer for a real persona — but it is not measurable in this harness, and pretending otherwise by driving it through its HTTP API would measure something no user of it does.

Transformer and LLM matchers. Need a GPU to be measured at a size that means anything, and the research code in this category is largely unmaintained. Covered as a category position, at cited.

Two disciplines#

Every number in S2 carries its conditions. Machine, architecture, interpreter, package versions, dataset, and the operating point. A precision figure without the threshold that produced it is not a measurement.

The harness is committed and re-runnable. Pinned versions, scripts that print to stdout, raw JSON in results/. A re-run a year from now can be diffed against what this survey claims, which is the only version of “reproducible” that survives the authors losing interest.

S2 verdict — how they differ, technically#

The measurements changed the shape of the answer#

S1 expected the comparison to be between matching philosophies: probabilistic against active learning against toolkit. That comparison exists and it is narrower than it looks. Splink and recordlinkage implement the same 1969 model with the same estimation procedure and produced identical results on FEBRL1 and results 0.7 points apart on FEBRL3. Two libraries with different architectures, different maintenance and a 17× speed difference agree on the answer, because the answer comes from the model and the model is the same.

What separated the candidates by more than that was everything around the model.

The three findings, ordered by how much they should change what you do#

1. Blocking outweighs the matcher. Loosening two blocking keys — same data, same comparison functions, same classifier — moved recordlinkage’s exact entity recovery from 0.894 to 0.970. The gap between the best and second-best matcher at their best settings is 0.7 points. Blocking recall is a hard ceiling, it is measurable before any model exists, and it costs one join to check. Spend the first hour there.

2. A model earns its keep by making better blocking safe. The same loosening that gained the learned models 7.6 and 1.8 points made the SQL baseline worse, 0.740 → 0.731. Extra candidates arrive as a mixture of true and false pairs; a model re-weights and absorbs them, a hand-picked constant cannot. This reframes what the probabilistic machinery is for. It is not primarily that it scores better on a fixed candidate set — it is that it decouples the blocking decision from the threshold decision, so you can improve one without redoing the other.

3. The pairwise score is not the output. Every run produces two numbers and the field reports the flattering one. Pairwise F1 0.815 corresponds to 73.1% of entities exactly right and 2,511 entities where there are 2,000. Worse, the entity count is not a validation: recordlinkage returned exactly 2,000 entities on FEBRL3 while getting 10.6% of them wrong, and Splink overshot by 3.95% while getting 4.1% wrong. A false pair costs one unit of pairwise precision and destroys two entities, so clustering amplifies precision errors and merely accumulates recall errors.

That asymmetry has a direct tuning consequence: prefer precision, then widen the blocking. It is the opposite of the instinct that lowers a threshold to catch more duplicates.

How the candidates actually differ#

Splink is the only one that does the whole job in one place, and its clustering is ordinary. Measured, its own clustering produced partitions identical to naive connected components at all four thresholds. Its advantage is pair precision — 0.999 into connected components beats 0.965 into the same algorithm by 6.5 points of entity recovery. The sophistication is upstream, in term-frequency adjustment and EM-estimated weights; the clustering is convenience, done correctly and at whatever scale the backend supports.

Its most underrated property is threshold insensitivity: 0.5 to 0.99 moved exact recovery by 0.4 points, against 8 points for the SQL baseline over a comparable sweep. A reader who picks badly loses almost nothing.

recordlinkage is the same model as a pandas pipeline, and it stops one step early. Its comparison-vector DataFrame is the best debugging artifact in the category — every approach here reduces to “produce a comparison vector, then classify it”, and this is the only candidate that hands you the vector. It returns matched pairs and offers no opinion on how many entities they imply, so the caller performs the transitive closure and inherits the amplification.

dedupe finishes the job further than anything else and cannot be relied on to run. It is the only candidate that canonicalizes — its canonical record recovered a date of birth one input lacked — and the only one whose clustering is hierarchical rather than a transitive closure. It learned better blocking predicates than the author wrote (recall 0.998 against 0.980) and scored the survey’s best result on FEBRL1. It also raised MemoryError on 5,000 records under a 6 GiB cap during prepare_training, and a fresh install raises AttributeError at blocking time because BTrees 6.5 removed a method it calls.

The SQL baseline is closer than the category admits and further than it feels. 73.1% exact entity recovery in 0.29 seconds and thirty lines is not a strawman. Splink’s 97.7% on the same data is 24.6 points better, which is the difference between a result you can publish and one you cannot. Both facts should be in front of a reader before they choose.

What S2 could not settle#

dedupe’s hierarchical clustering against connected components on hard data. On FEBRL1 the difference was small and in the predicted direction (99.6% against 99.2%). The regime where it should matter is long chains, which is FEBRL3, and FEBRL3 is where dedupe ran out of memory. This is the most interesting comparison the survey could not complete and it is recorded as a gap rather than guessed at.

Whether any of this holds on non-personal data. FEBRL is synthetic Australian personal records. The mechanisms generalize — blocking sets a ceiling, a constant threshold cannot absorb extra candidates, clustering amplifies false pairs — because they follow from how the tools work. The absolute numbers do not, and the survey makes no claim that they do.

Scale. 5,000 records is not a scale test. What can be said about scale is architectural and is said as architecture, at cited.

What proceeds to S3#

The technical picture is settled enough that the remaining question is not “which is best” but “who is choosing, and under what constraints”. Four things S2 measured turn out to be persona-defining rather than tool-defining: whether labels can be produced at all, whether the data may leave the building, whether anyone will inspect the output, and whether the job runs once or every week. S3 takes those.

recordlinkage — architecture and measurements#

The three-object pipeline#

The design follows scikit-learn, and that is its main contribution: it makes record linkage look like a machine-learning pipeline, with every intermediate artifact inspectable.

Index produces candidate pairs. block on one or more columns, sortedneighbourhood for a sliding window over a sorted key, random for sampling, full for the quadratic version. Rules are unioned. The output is a pandas MultiIndex.

Compare turns each candidate pair into a comparison vector — one column per field comparison, each between 0 and 1. string with jarowinkler, levenshtein, damerau_levenshtein, qgram or cosine; exact; numeric with several curve shapes; date; geo. The output is a DataFrame, one row per candidate pair.

That DataFrame is the reason the library is worth reading even if you never ship it. Every approach in this category reduces to “produce a comparison vector, then classify it”, and recordlinkage is the only candidate that hands you the vector as an ordinary pandas object you can plot, sort and inspect. Debugging a linkage means looking at the vectors of the pairs that went wrong, and here that is a .loc.

A classifier turns vectors into matches. Supervised: LogisticRegressionClassifier, NaiveBayesClassifier, SVMClassifier. Unsupervised: KMeansClassifier and ECMClassifier.

ECM is the one that matters#

ECMClassifier is expectation-maximization over the comparison vectors, estimating the same m and u probabilities Splink estimates, from the same Fellegi-Sunter model. It requires no labels. It is scored here for that reason: it is the fair comparison against Splink’s EM and against a baseline that gets no labels either.

It takes a binarize parameter, because the classical Fellegi-Sunter model expects categorical agreement levels and the comparison vectors are continuous. The probe used binarize=0.85 — anything above 0.85 similarity counts as agreement. That is a real modeling choice made by the caller, and Splink’s equivalent is choosing the thresholds inside each comparison; neither library estimates it.

Measured#

FEBRL3, prefix blocking, aarch64, CPython 3.12.3.

Blocking 0.046 s, comparison 0.109 s, ECM fit and predict 0.11 s.

FEBRL1: F1 0.989, 511 entities against 500, 97.8% exactly right — identical to Splink’s result on the same data, in 0.03 seconds against Splink’s 1.22.

The two things this table settles#

The model beats the no-model threshold, and by a lot. ECM at F1 0.975 against 0.934 for the best fixed sum, and 97.0% entity recovery against 92.5%. The sum-of-vector rows are the same comparison vectors with the modeling removed, so the difference isolates what EM contributes: about 4.5 points of entity recovery here, and it is free.

recordlinkage and Splink implement the same model and get almost the same answer. On FEBRL1 the two are identical to three decimal places. On FEBRL3 Splink is ahead by 0.7 points of exact recovery. This is what should happen — they are both Fellegi-Sunter with EM — and it means the choice between them is not really about accuracy.

It is about everything else: recordlinkage is 17× faster on this data and holds everything in pandas; Splink pushes into a SQL engine, scales to backends recordlinkage cannot reach, clusters for you, and is being maintained.

Where it stops#

ECMClassifier.fit_predict returns a MultiIndex of matched pairs. That is the end of the library.

The caller must then decide how many entities those pairs imply. Every partition reported for recordlinkage in this survey came from the harness running connected components, and that step is where a single false pair welds two true entities together — the amplification described in clustering-and-the-pairwise-illusion.md. Under configuration A, where recordlinkage’s precision was 0.965 against Splink’s 0.999, the 3.4-point precision gap became a 6.5-point entity-recovery gap.

The library is a toolkit and this is its documented scope. The missing step is the expensive one to get wrong.

Dependencies and durability#

pandas (<3), numpy, scipy, scikit-learn, jellyfish, joblib. Pure Python, any wheel, installs in 1.7 seconds with no compiler on both 3.12 and 3.13 — better install behavior than several maintained candidates.

The exposure is the pandas ceiling. pandas<3 is a constraint a maintained library would lift; an unmaintained one cannot. Until pandas 3 is widespread this costs nothing, and after it does, recordlinkage stops being a component that upgrades and becomes a constraint on everything else in the environment. That is the mechanism by which “unmaintained but working” turns into “unmaintained”, and it has a date attached that nobody controls.

Splink — architecture and measurements#

The model#

Fellegi-Sunter, 1969. For a candidate pair, each compared field falls into an agreement level — exact, close, different — and each level carries two probabilities:

• m: the probability of observing that level given the pair is a match
• u: the probability of observing it given the pair is not a match

The match weight for a field is the log ratio m/u, and the weights sum across fields into a total that converts to a match probability. The structure is what makes the approach worth its complexity: agreement on a rare surname and agreement on a common one produce different weights, because u differs, and that difference is estimated from the data rather than chosen. Term-frequency adjustment is this idea applied per value, and it is the single largest thing a probabilistic engine offers over a hand-weighted score.

How the parameters get estimated without labels#

Three steps, none of which requires a labeled pair, and the probe ran exactly these:

u by random sampling. Draw random pairs. Almost all are non-matches, because matches are vanishingly rare among all pairs, so the observed agreement rates approximate u directly. The probe sampled 2,000,000 pairs.

m by expectation-maximization, run on a blocked subset. Blocking on one field makes matches dense enough for EM to separate the two populations, and the field blocked on is excluded from that pass because its agreement is now constant. Two passes were run — blocked on date of birth, then on surname plus given name — so that every comparison gets an estimate from at least one pass. EM converged in 3 iterations.

The prior, probability_two_random_records_match, estimated from a set of blocking rules plus a stated recall assumption.

Total training time on FEBRL3: 2.06 seconds. That is the whole cost of the “unsupervised” claim, and it is the reason it is the default recommendation — the expensive input in an entity-resolution project is labeled pairs, and this removes it.

The caveat: unsupervised is not unconfigured. The reader still chooses which fields to compare, which comparison function and thresholds for each, and which blocking rules. Those choices moved the result more than the estimated parameters did — see blocking.md.

SQL generation, and what it buys#

Splink does not compare records in Python. It builds SQL and hands it to a backend, going through sqlglot for dialect translation. DuckDB is the default and a hard dependency; PySpark, Athena, SQLite and PostgreSQL are extras.

The consequences are concrete:

The data never enters Python. On the DuckDB backend the comparison runs in a vectorized C++ engine over columnar data, in-process. That is why 20,860 candidate pairs across six comparisons take 0.185 seconds.

The model is portable and the performance profile is not. The same settings object generates Spark SQL. What does not carry over is tuning: partitioning, shuffle behavior and salting are Spark concerns that do not exist on DuckDB. The design point is that the model survives a backend change, and that is a real and unusual property — but a team migrating should expect to tune the new backend, not to inherit the old one’s speed.

Debugging can mean reading generated SQL. When something is slow, the artifact is a query, not a stack frame.

Measured#

FEBRL3, 5,000 records, 2,000 true entities, prefix blocking, aarch64, CPython 3.12.3.

Training 1.71 s, prediction 0.185 s, clustering under 0.1 s per threshold.

FEBRL1, 1,000 records: F1 0.989, 511 entities against 500, 97.8% exactly right.

The threshold barely matters, which is the most reassuring property in the table. Between 0.5 and 0.99 the entity count moves by 14 out of 2,000 and exact recovery by 0.4 points. The match probabilities are well separated, so a reader who picks a threshold badly loses almost nothing — and that is a meaningful practical advantage over the SQL baseline, whose threshold choice moved exact recovery by 8 points across the same kind of sweep.

What it does not do#

It does not canonicalize. It returns a cluster ID per record. Which of the six spellings is the right one, and what to write into the surviving row, is the caller’s problem. dedupe is the only measured candidate that answers it.

Its clustering is connected components. Measured: identical partitions to the naive transitive closure at all four thresholds. The advantage is upstream pair precision. See clustering-and-the-pairwise-illusion.md.

Version 4 broke version 3’s API. The migration is documented; the internet is not. A newcomer searching for Splink examples will find a majority written against Splink 3, and they do not run.

The SQL baseline — architecture and measurements#

Scored on the same benchmark, at the same operating points, as the libraries. It is the thing a library has to beat, and it is what most teams ship.

The two candidates#

B1 — exact match on a normalized key. Lowercase and trim the name fields, strip non-digits from the date, group on the concatenation. Blocking, deciding and clustering are all the same GROUP BY. Five lines.

B2 — a blocked similarity join. Three blocking conditions unioned into a self-join; a hand-weighted sum of four Jaro-Winkler comparisons (0.30 given name, 0.30 surname, 0.25 date of birth, 0.15 suburb); a threshold; connected components over the accepted pairs. About thirty lines.

Neither is a strawman. B2 blocks on three keys, treats fields differently, and handles the transitive closure — it is a fair description of a competent afternoon, and it is the version of “just write the SQL” that a reader should be comparing against.

What the engines provide#

DuckDB 1.5.5, verified against the installed build: jaro_similarity, jaro_winkler_similarity, levenshtein, damerau_levenshtein, jaccard, hamming, editdist3, strip_accents. No phonetic function — no soundex, no metaphone. In-process, reads CSV and Parquet directly, 52.6 MiB, one distribution.

PostgreSQL, via two extensions in the standard distribution: pg_trgm for trigram similarity with GIN and GiST index support, and fuzzystrmatch for levenshtein, soundex, metaphone and daitch_mokotoff_soundex. The index support is the real differentiator — a trigram similarity join can use an index, which changes the scaling story rather than just the constant.

SQLite, via FTS5 for blocking and editdist3 in spellfix1. fuzzymatcher is built on exactly this.

Measured — FEBRL3#

5,000 records, 2,000 true entities, prefix blocking, DuckDB 1.5.5, aarch64.

Blocking: 8,818 candidate pairs from 12,497,500 — a reduction ratio of 0.99929 — retaining 96.1% of true pairs.

FEBRL1: B2 at 0.80 reaches F1 0.841 and 72.0% exact recovery.

Four things this establishes#

The bar is not low. Thirty lines of SQL, no dependency, 0.29 seconds, and 73% of entities exactly right. Any library that cannot clearly beat that has not earned its place in a requirements file, and two of the candidates in this survey cannot even be installed.

The bar is not high enough either. Splink reached 97.7% on the same data. The gap — 24.6 points of exact entity recovery — is what the probabilistic machinery is worth, and it is large. For a dataset where 27% of entities are wrong versus 2.3%, that is the difference between a result you can publish and one you cannot.

Exact matching fails in the shape that passes review. B1 scores precision 1.000: nothing was wrongly merged, and anyone spot-checking the merges will find zero errors. It returns 3,682 entities where there are 2,000 — 84% too many. All the error is in the half that never merged, which is the half nobody samples. This is the single most common way a deduplication is declared successful.

Tightening the threshold makes it monotonically worse. From 0.80 to 0.92, precision rises 0.954 → 0.999 and the entity count moves 2,511 → 3,031, further from the true 2,000 at every step. The instinct that produces this — raise the threshold until the bad merges stop — optimizes the number a reviewer looks at and degrades the output.

The ceiling, and why it is structural#

It cannot learn that a rare surname is worth more than a common one. The weights (0.30/0.30/0.25/0.15) were chosen by hand and are wrong in a way that is invisible without ground truth. Term-frequency adjustment — weighting agreement by how rare the agreed value is — depends on the data’s own distribution and cannot be written as a constant. It is the largest single thing the probabilistic engines add.

It has no principled way to choose the threshold. 0.80 is a guess that survived a histogram. The sweep above shows the choice is worth 8 points of exact recovery, and nothing in the SQL tells you which to pick.

It cannot absorb better blocking. This is the finding that most changes the recommendation. Loosening the blocking rules raised blocking recall from 0.932 to 0.961 and moved the baseline’s exact recovery down, 0.740 → 0.731, while moving Splink’s up 1.8 points and recordlinkage’s up 7.6. A fixed threshold converts extra candidates into false positives; a model re-weights and absorbs them. The baseline is therefore locked to the blocking it was tuned against, and improving either requires redoing both.

When the baseline is still the right answer#

When the join key is nearly clean and B1’s recall is acceptable. When the dataset is small enough that a person will check the output. When adding a dependency to the environment costs more than the 20 points of accuracy is worth — which is a real situation, and more common in this category than the library authors’ documentation suggests. And always as the first measurement, because it takes ten minutes and it tells you how hard your actual problem is before you choose anything.

S3 approach#

What S2 handed over#

The technical comparison came out narrower than expected. Splink and recordlinkage implement the same model and produced identical results on one dataset and results 0.7 points apart on the other. Blocking moved the outcome more than the choice of library. The clustering algorithm in every measured candidate but one is connected components.

So “which library is best” is close to settled and close to unimportant. What actually varies between real situations is not the algorithm. It is four constraints, each of which S2 measured and none of which is a property of any tool:

1. Can labeled pairs be produced at all? Somebody has to sit and answer “same person?” a few dozen times. In some situations that person exists and knows the data; in others there is nobody who could answer.
2. May the data leave the building? This eliminates a whole column of options before any evaluation, and it eliminates it first.
3. Will anyone look at the output? A cluster nobody inspects has to be right. A cluster somebody inspects only has to be a good suggestion, which is a much easier problem and admits a completely different tool.
4. Does it run once, or every week? A one-off cleanup and a recurring pipeline want opposite things, and the mistake runs in both directions.

The personas below are organized by those constraints rather than by industry, because two organizations in the same industry with different answers to question 3 need different tools, and two in different industries with the same answers need the same one.

Where the personas came from#

Three are drawn from situations that produce this problem repeatedly and are documented well enough to describe accurately: a membership organization reconciling its roster against an events feed, a migration off a long-lived desktop database, and a collection held in several catalogs that have drifted apart.

Three more cover the constraint corners the first three leave empty: continuous resolution where records keep arriving, a one-off cleanup by somebody who will not write a program, and matching public datasets where the merge has to survive challenge.

Each is written as who and why. What they should reach for is stated at the end and the reasoning is in S2; none of these files explains how to implement anything.

The one thing every persona shares#

Spend the first hour on blocking and normalization. It is the finding that survived every framing S2 tried, it outweighs the choice of library in every persona below, and it is the step that is skipped most often because it does not feel like the interesting part of the problem.

S3 verdict — who should use what#

The table#

Four constraints did more work than the tool comparison#

Whether labels can be produced. Active learning scored the survey’s best result on the dataset where it ran — dedupe, F1 0.998 from 40 labels. It is also the wrong choice for the migration persona (runs once, model discarded) and unavailable in the collection persona (nobody can answer without the object in front of them). The technique’s quality and its applicability are close to unrelated.

Whether anyone will inspect the output. Two personas here — the spreadsheet and the collection — have a domain expert confirming each merge, and both are best served by a desktop application this survey would otherwise have no reason to cover. The other four have nobody checking, which raises the accuracy bar and rules OpenRefine out. This one constraint splits the field cleanly and is not a property of any tool.

Whether the data may leave. It removes the managed column outright in the public-interest persona and often in the continuous one, before any evaluation.

Whether it runs once or forever. The migration persona should take on no dependency; the continuous persona needs stable entity IDs, which is a requirement no benchmark in this survey measures and which disqualifies the best-scoring tool as-shipped.

Two things every persona shares#

Blocking and normalization first. In six different situations, with six different tools, the same advice. It moved the measured result more than the choice of library did — 7.6 points of entity recovery against 0.7 between the top two candidates — and it is the step most often skipped, because it looks like preparation rather than work.

Under-merge by default. A false merge destroys two entities and is often irreversible downstream; a missed merge leaves two rows that can be joined later. This asymmetry is mechanical — it follows from how connected components propagates a false edge — and it points the same way in every persona, including the ones where the instinct is to catch as many duplicates as possible.

Where the personas disagree, and why that is the point#

On precision versus recall. The public-interest persona wants precision 1.000 and will pay a point of recall for it. The membership persona wants recall, because a member who does not appear to attend is a wrong number in a board report. Same tools, same measurements, opposite operating points — and a survey that recommended one threshold would have been wrong for one of them.

On whether a library is the answer at all. Two of six personas should use a desktop application, and one should mostly use SQL. Half of this category’s real situations are not solved by the thing the category is named after, which a survey of libraries should say out loud.

On which failure is affordable. The collection persona can live with a 20% wrong suggestion rate because a curator filters it. The continuous persona cannot live with a 1% wrong merge rate because it acts on the result automatically. Neither is being careless; they are absorbing error in different places.

What S3 hands to S4#

The technical and situational pictures both point at Splink for the general case and at non-library answers for two of six specific ones. What neither pass has examined is whether these choices survive: dedupe was the category’s default for a decade and now raises on a fresh install, recordlinkage is the most-installed library here and has no maintainer, and the healthiest project in the survey is a Java desktop application. S4 takes durability.

Who Needs This#

An institution whose collection is described in several catalogs that have drifted apart, and which needs one list of what it actually holds.

A museum, a historical society, an archive, a library special collection, a corporate heritage department. There is a catalog from the 1990s, a spreadsheet somebody kept in parallel, an export from a collections-management package, and a set of accession registers that were partly transcribed. Each was authoritative at some point. None is now.

The person doing the work is a collections manager or a volunteer with a background in the subject rather than in data, sometimes with a consultant for a few weeks.

Why They Need It#

This is the hardest persona in the survey, and the reason is the data#

Everything that makes personal-name linkage tractable is absent.

There is no date of birth. The single most discriminating field in the classic record-linkage problem does not exist. An object has an accession number if it is lucky, a date range if it is not, and “circa 1890” if it is typical.

The titles are descriptions, not names. “Portrait of a woman in a blue dress”, “Woman in blue, portrait”, and “Unidentified woman (blue dress)” may be the same object or three objects. A string metric has no opinion worth having about that, and neither does a person without the object in front of them.

The identifiers are the thing that drifted. Accession numbers were reformatted in a system migration. One catalog prefixes the year, another does not. A third has them in a free-text notes field because the import dropped the column. The field that should carry the join is often the field most damaged by the history that created the problem.

Sameness is genuinely ambiguous. Are a photograph and its negative one object or two? A set of six chairs? A letter and its transcription? The catalogs disagree, and they disagree because their compilers had different and defensible answers. No matching algorithm resolves a question the institution has not answered.

The constraint that changes everything#

Somebody will look at every merge, and that person is a subject expert.

This inverts the requirements. In most personas the tool has to be right because nobody will check. Here the tool has to produce good suggestions, ranked, with the evidence visible — and being wrong 20% of the time is acceptable if the expert can see why in two seconds and reject it.

That changes which tool wins. An interactive clustering interface a curator can drive beats an accurate batch process they cannot inspect, because the curator’s judgment is the part that cannot be automated and the bottleneck is how fast they can exercise it.

Why the automated approach underperforms here specifically#

Blocking is the step this survey found matters most, and it is the step this data frustrates. Blocking needs a field that is cheap to compare and usually agrees for true matches: a date of birth, a postcode, a surname. Object catalogs have no such field. Blocking on the title means blocking on the thing that varies most; blocking on the accession number means blocking on the thing that was corrupted.

The measured consequence is predictable from the mechanism rather than from a benchmark — this survey’s datasets are personal records, and it does not claim otherwise. But the mechanism is clear: blocking recall sets a hard ceiling, and this persona’s blocking options are all poor, so the ceiling is low before any model runs.

What actually helps#

Answer the sameness question first, in writing. What counts as one object, for this institution, this year. This is a cataloging decision, not a technical one, and no tool substitutes for it. Projects that skip it produce clusters nobody can adjudicate because the adjudicator does not know the rule.

Reconcile pairwise, not all at once. Four catalogs is six pairs, and each pair has its own conventions and its own best blocking key. Merging all four into one pile and running one resolver discards the structure that makes each pair tractable.

Use the accession number where it survives, and repair it where it does not. A normalization pass over identifier formats is worth more here than any matcher, exactly as in the migration persona. Whatever fraction joins on a repaired identifier is a fraction nobody has to adjudicate.

Treat the residue as a queue, not a failure. The output that helps a curator is a ranked worklist with the evidence shown, worked through over months.

What to reach for#

OpenRefine, first and possibly only. It is the best fit in the survey for this persona: it clusters values with a person confirming each merge, its keying functions (fingerprint, n-gram fingerprint, phonetic) are well suited to title and maker-name variation, and the curator drives it directly rather than through a programmer. It is also the healthiest project in this survey by a distance. Its ceiling — one column at a time — is a real limitation and is less binding here than elsewhere, because the fields that would combine into evidence are mostly missing anyway.

Splink for the identifier-and-maker join, if there is a programmer and enough structured fields to combine — maker, date range, medium, dimensions. Use it to produce ranked suggestions at a high threshold, not decisions.

A reconciliation service for names of people and places. Matching a maker against an external authority is a different and much easier problem than matching two internal catalogs, because the authority is stable and curated. It is the one place in this persona where automation is reliable.

Not an unattended batch pipeline. The expert judgment is the asset. A tool that removes the expert from the loop has removed the only thing that can answer the questions the data actually poses.

Who Needs This#

A team where records keep arriving and the entity list has to stay correct between arrivals.

A customer data platform, a patient index, a supplier master, a sanctions-screening pipeline, a fraud team. Several sources write continuously. Downstream systems hold references to entity IDs and act on them — send the email, deny the transaction, merge the chart.

The person doing the work is a data engineer who owns this as a service, not as a project.

Why They Need It#

Batch resolution has a property that breaks this persona#

Every candidate this survey measured resolves a dataset. Run it again on a superset and you get a new partition — and the entity IDs are not the old ones. Connected components assigns identifiers by traversal order, and the traversal changed.

For a one-off deduplication that does not matter. For a service whose IDs are foreign keys in five other systems, it is the whole problem. Yesterday’s entity 4471 is today’s 8802, and everything that referenced it is now wrong.

This is the requirement that separates this persona from every other one in the survey, and it is not a matter of accuracy. A tool that scores 97.7% and reshuffles its IDs nightly is unusable here, and a tool that scores 90% with stable IDs is not.

The second requirement: merges have to be reversible#

In a batch job a wrong merge is a wrong row in an output nobody has acted on. Here, a wrong merge means two customers’ order histories were combined, an email went to the wrong address, or two patients’ records were joined. Somebody will find out and the merge will have to be undone — after downstream systems have already acted on it.

That means the system must record why each merge happened and must be able to split a cluster without recomputing everything. Very little in this category offers either.

The third: latency#

A record arriving has to be resolved in the time budget of the request that carried it, not in the next nightly window. Re-running a full resolution over the whole dataset per arrival is not an option at any size, and it stops being an option quickly.

What the survey’s measurements say about this#

Bluntly: this is the persona the open-source tier serves worst.

Splink — the best measured result here — is a batch process. Its clustering is connected components over a threshold, recomputed each run. There is a documented pattern for scoring new records against a stored model, which handles the latency question, but the cluster-identity question is the caller’s to solve, and solving it well is a real piece of engineering: a stable ID store, a merge log, and a policy for what happens when new evidence links two previously-separate clusters.

dedupe offers a Gazetteer shape, matching new records against an already-resolved set, which is closer to what this persona needs. It also has 14 patches waiting with none merged in six months and raised MemoryError on 5,000 records in this survey’s harness.

Senzing is the tool designed for exactly this, and its architecture is the reason it belongs in the survey despite not being measurable in it: it resolves incrementally by design rather than by re-running, it maintains entity identity across arrivals, and it can explain and unwind a merge. That is a different product category from a batch linkage library, and this is the persona where the difference is the point rather than a detail.

The cost is stated plainly in its own file: the SDK carries an Apache-2.0 identifier and the engine is licensed separately, pip install senzing-core succeeds and then raises because the engine is not there, and the free license covers 500 records — below the smallest dataset in this field’s own benchmark family. Evaluating it is a procurement step, not an afternoon.

What goes wrong#

Building the batch version first and discovering the ID problem in production. It is the standard path into this persona and it is expensive, because by then other systems hold the IDs.

Treating a merge as final. Every real system in this shape eventually needs to split a cluster, and a design that cannot is a design that will be worked around with manual SQL at some point, by someone in a hurry.

Optimizing accuracy before stability. The measurements in this survey rank tools by accuracy because that is what a benchmark can measure. For this persona accuracy is the second question and identity stability is the first.

What to reach for#

Senzing, if a vendor relationship and a licensed engine are acceptable and the data may stay where the engine runs. It is the only candidate in the survey whose architecture matches the requirement rather than being adapted to it.

Splink plus your own identity layer, if it must be open source. Store the model, score arrivals against it, and keep entity IDs in a table you own with an explicit merge log — accepting that the identity layer is a component you are building and maintaining, not one you are adopting. The linkage part is solved and excellent; the part this persona actually needs is the part you write.

Zingg, if the team is already on Spark and can satisfy AGPL-3.0. It documents incremental runs against an existing entity set, which is the right shape. The license is a decision for someone with authority to make it, before the technical evaluation.

Not a nightly full re-resolution with regenerated IDs. It is the design this persona arrives at by default and it is the one that has to be replaced later, under pressure.

Who Needs This#

Anyone moving a database that has been in daily use for fifteen or twenty years onto a new schema.

A desktop database — Access, FileMaker, dBase, a FoxPro application somebody’s uncle wrote — that has outlived its platform and every convention it started with. It works. It has been worked around. The migration is happening because the platform is ending, not because anyone wanted it.

The person doing the work has a target schema they designed, a source they did not, and a deadline set by something outside their control.

Why They Need It#

Most of this job is not entity resolution, and mistaking it for entity resolution is#

the expensive error

The instinct on opening a twenty-year-old table is that it needs a matching engine, because the data is visibly messy. Usually it does not. It needs parsing, normalization and a mapping table, and the truly ambiguous remainder is small.

The order matters enormously. Normalizing first and matching second turns a hard matching problem into an easy one; matching first means asking a model to discover conventions that could have been written down in an afternoon. “8236 SE 24th St” and “8236 Southeast 24th Street” have a Jaro-Winkler similarity that punishes them for a convention. Parsed into components, they agree exactly, and no threshold tuning ever recovers what normalization would have given for free.

What is actually in the source#

Conventions that changed and were never backfilled. The organization switched from “LASTNAME, FIRSTNAME” to “Firstname Lastname” in 2009 and both are in the table. Dates are in three formats because three people entered them. There is a status column whose values include A, Active, active, ACT and, twice, Actve.

Fields used for something other than their name. A notes column that became a de facto category. A phone field with an email in it. This is not entity resolution either — it is archaeology, and it needs a human reading a hundred rows.

Genuine duplicates, of a specific kind. Old databases accumulate duplicates from re-entry rather than from variation: someone could not find the existing record and made a new one. Those duplicates are often near-identical on the fields that matter, which means an exact or near-exact key catches most of them. It is a much easier deduplication problem than a merge of two independently-maintained systems.

The three constraints that define this persona#

There is a target schema, and it is the point. Unlike every other persona here, the output is not “which records are the same” — it is a populated new database. Resolution is one stage in a pipeline whose other stages are mapping, type coercion and validation, and it should be sized accordingly.

The migration happens once. Nothing recurring justifies a dependency that has to be maintained for years. This argues strongly for tools already present, and for SQL over a library.

The result has to be defensible to people who remember the old system. Somebody will look up a record they know by heart and check it survived. That means the mapping needs to be inspectable, not just correct — a lookup table somebody can read beats a model somebody has to trust.

What goes wrong#

Reaching for a matching engine on day one, before normalizing, and spending a week configuring comparisons that a parsing pass would have made unnecessary.

Deduplicating too aggressively, on the theory that a clean target schema should have no duplicates. Two records that look identical may be two real people, and a migration that silently merges them has destroyed data that the old system, for all its faults, was holding correctly. The default posture should be to under-merge and flag, because under-merging is recoverable later and over-merging is not.

Treating the ambiguous remainder as a failure. After normalization there will be a few hundred rows that cannot be resolved automatically. That is not a defect in the process — it is a worklist, and it is finite, and somebody who knows the organization can work through it. Turning an unbounded risk into a bounded task list is a successful outcome.

What to reach for#

SQL first, and probably SQL last. Normalize into a staging table, join on the normalized key, and look at what does not join. Measured on the closest comparable benchmark, exact matching on a normalized key achieved precision 1.000 — it merges nothing wrongly, which is exactly the posture this persona wants — and the records it leaves unmatched become the worklist rather than a silent error.

Parsing libraries, chosen for the domain. Names and addresses have dedicated parsers that encode conventions nobody should re-derive. This is where the effort belongs.

A resolver only for the remainder, if the remainder is large enough to justify one. Splink on the unmatched residue, with a high threshold, producing suggestions for a human to confirm — not decisions to apply.

Not an active-learning tool, in most cases. The labeling effort is a fixed cost, the job runs once, and the model is discarded afterwards. That is the worst possible return on labeling.

Who Needs This#

An organization that keeps a roster of members and a separate record of who attended what, and needs to know which attendee is which member.

A professional association, a chapter of a national body, a club, a union local, a congregation. There is a membership system with a few hundred people in it, and there is an events or registrations system with several thousand rows because people register more than once. The two systems were bought years apart, they do not share an identifier, and nobody has ever joined them.

The person doing the work is usually the one competent programmer in an organization of volunteers, and they are doing it on a Saturday.

Why They Need It#

The question is not academic#

Everything the organization would like to know sits behind this join. Which members show up and which have not been seen in two years. Whether the people filling the room are members at all. Whether the renewal rate differs between attendees and non-attendees. Every one of those is a GROUP BY away if the join exists, and unanswerable if it does not.

Why the join is hard, specifically#

The two systems asked for the name differently, and both let people type.

The roster holds “Last, First” because it sorts. The registration form asked for a display name, so it holds “First Last” — and, because these are professionals, often “First Last, CSP” or “First Last, PhD” or “First Last, MBA, PMP”. A registration made by an assistant carries the assistant’s email. A registration made three years ago carries a maiden name. Somebody typed a nickname once and it stuck.

None of this is unusual and all of it is invisible until the join fails.

The three things that make this persona distinctive#

The asymmetry is large and it helps. A few hundred members against several thousand registrations. That means the expensive side is small, and it means the correct framing is link, not dedupe: every registration belongs to at most one member, and members are the authority. The one-to-one structure is real information and a linkage tool can use it.

Somebody in the building knows every answer. This is the persona’s biggest advantage and it is routinely wasted. The membership secretary can look at two rows and say “yes, that’s her, she remarried” in under a second, and can do it three hundred times in an afternoon. Very few situations in this category have an oracle sitting down the hall.

Nobody will inspect the output. The join gets built once and then feeds a dashboard or a mailing list forever. That raises the bar: a cluster that is 90% right is not a good suggestion here, it is a wrong number that will be quoted at a board meeting.

What goes wrong when they do it themselves#

The default attempt is an exact join on lowercased name, and it matches perhaps a third of the rows. Measured on the closest available benchmark, exact matching on a normalized key recovered 31% of true pairs — and reported precision 1.000, because everything it did match was right.

That combination is the trap this persona falls into hardest. The join looks correct. Every matched pair spot-checks clean. The two thirds that did not match are invisible, and the resulting report undercounts member attendance by a factor of three while appearing to be conservative and careful.

The second attempt adds a fuzzy comparison on the name and a threshold, and produces the opposite failure: two members with the same common surname get merged, and the dashboard now shows one person who attended fourteen events.

What actually helps#

Parse before matching. Splitting “First Last, CSP” into components and stripping the credential suffix does more here than any model. So does deciding, once, that the roster’s “Last, First” and the registration’s “First Last” are the same shape. This is the normalization step, it is boring, and in this persona it is most of the work.

Use a second field. Email is the obvious one and it is worth more than the name, because it is either right or absent rather than approximately right. Postcode and joining year help. The whole point of a record-linkage tool over a string comparison is that it weights fields by how much evidence they carry, and this persona usually has two or three fields available and uses only one.

Ask the secretary. The active-learning tools exist for exactly this shape — a person who can answer, a library that picks the questions. Measured, forty answered questions produced the best result in this survey on the comparable dataset.

What to reach for#

Splink, link_only, with email and postcode alongside the parsed name. It needs no labels, so the Saturday does not have to include a labeling session; it clusters, so the output is member IDs rather than pairs; and it runs on DuckDB in-process, which matters because this organization has no data infrastructure and is not going to acquire any.

Consider dedupe instead if the secretary is available and willing, and pin BTrees<6.5 before starting. Its active learning is the best fit for this persona of anything in the survey, and the constraint is not technical: it is whether an hour of the one person who knows the answers can be booked.

Do not reach for a fuzzy string matcher on the name alone. It is the tool this persona finds first, it appears to work, and it cannot distinguish two members who share a surname — which, in a professional association, is the case that will be noticed.

Who Needs This#

Somebody joining two public datasets to make a claim that will be disputed.

A reporter matching a lobbyist register against a list of contract awards. A researcher linking court records to an address file. An oversight group checking whether board members of a public body also hold interests in its suppliers. An auditor.

The output is not a dashboard. It is a sentence in a published piece — “forty-one of the contracts went to companies with a director in common with the awarding body” — and somebody with a lawyer is going to read it.

Why They Need It#

The requirement is not accuracy, it is defensibility#

An entity-resolution result that is 97% right is excellent by any benchmark in this survey and useless here if the 3% cannot be characterized. The question that will be asked is not “how accurate is your method” — it is “why did you say this specific person is that specific person?”, about one row, in public.

That reorders everything.

Every merge must be explicable individually. Not the model’s overall precision: this merge, this pair, these fields. A tool that outputs a cluster ID and no reasoning fails the requirement even if it is more accurate than one that shows its working.

The method must be reproducible by a hostile reader. Somebody should be able to take the same two public files and the same code and get the same answer. That means pinned versions, committed code, and a stated threshold — and it rules out anything whose behavior depends on a vendor’s model that may change next quarter.

False positives are catastrophic and false negatives are merely disappointing. Naming the wrong person is a correction, or worse. Missing a match means the story is smaller than it could have been. These are not symmetric and no benchmark’s F1 reflects that.

What that means for the threshold#

Every other persona in this survey wants a balanced operating point. This one does not.

The measured threshold sweeps are unusually useful here because they show what tightening costs. Splink at threshold 0.99 reached precision 1.000 with recall 0.945 — against 0.999 and 0.956 at threshold 0.5. Between those two operating points, precision became perfect at a cost of about one point of recall.

For this persona that trade is obviously correct, against an instinct that in every other context sits at the balanced point. Here the right posture is a high threshold, an explicit “possible matches” tier below it that a human works through by hand, and a published count of both.

The SQL baseline behaves the same way and the sweep is steeper: precision 0.954 at threshold 0.80 rising to 0.999 at 0.92. If a hand-written SQL join is the method — and for a two-file public-records join it often should be, because it is the most legible method available — the tightened end of that range is where to sit.

Why the legibility of the method matters more than usual#

Whatever is published, somebody may have to explain it to an editor, a lawyer, or a committee. That audience can follow “we matched on exact company number where present, and on normalized name plus postcode otherwise, and hand-checked the 212 that matched on name alone”. They cannot follow an EM-estimated match weight, and neither, under questioning, can most of the people who ran it.

This is the one persona where a less accurate method can be the correct choice, because the marginal matches a probabilistic model finds are exactly the marginal matches that cannot be defended one at a time.

What goes wrong#

Publishing the aggregate without working the marginal cases. The forty-one contracts include six that matched at 0.86 and were never looked at, and one of those six is the one a subject’s lawyer will find.

Matching on name alone in a jurisdiction where the name is not distinctive. Common surnames make a name match nearly meaningless in some populations and nearly conclusive in others, and a method that treats them the same is wrong in a way that maps onto real groups of people.

Treating an official identifier as authoritative without checking it. Company numbers, license numbers and registration IDs are the best evidence available and they are also reused, reissued and mistyped in public files.

What to reach for#

A SQL join on the strongest available identifier, then a documented cascade of weaker rules, with counts published at each tier. It is the most defensible method available, it takes an afternoon, and its reasoning fits in a paragraph a reader can check.

Splink at a high threshold, when the identifier cascade is not enough. Its match weight waterfall is the reason to pick it here: for any individual pair it shows which field contributed what to the decision, which is exactly the artifact this persona needs when asked about one row. It is MIT, it pins cleanly, and it is maintained by a government analytical team whose own work is subject to public scrutiny — which is not an argument about quality, but does mean the diagnostics were built for people who get asked these questions.

A published method note and a committed harness. Pinned versions, the blocking rules, the threshold, and the counts at each tier. This survey’s own harness is built to that standard for the same reason.

Not a managed matching service, for two reasons that have nothing to do with quality: the data usually cannot be uploaded, and a result that cannot be reproduced without a vendor account cannot be checked by the person disputing it.

Who Needs This#

Somebody who was sent a spreadsheet, needs the same thing to stop appearing under four spellings, and is not going to write a program.

An analyst, an operations manager, a grant administrator, a journalist on a deadline. The file has a few hundred to a few thousand rows and one column that matters — supplier name, organization, job title, city, category. The task is due this week and is not part of a system.

This is the most common situation in the entire category and it is the one a survey of libraries is most likely to answer badly.

Why They Need It#

The job is smaller than the tooling assumes#

A pivot table by supplier is wrong because “Acme Corp”, “Acme Corporation”, “ACME Corp.” and “Acme Corp” (trailing space) are four rows. That is the whole problem. There is no schema, no pipeline, no second dataset, and no next month.

Every library in this survey assumes a Python program is being written. That assumption excludes this person entirely, and they outnumber every other persona here.

Why the obvious answers fail#

Find-and-replace works until there are ninety variants, which happens sooner than expected because the variation is combinatorial: punctuation, legal-form suffix, abbreviation, case, whitespace, and a typo.

Sorting and scanning works and is what most people actually do. It is also how a whole afternoon disappears, and it silently misses anything where the variation is in the first characters — “The Acme Corporation” sorts nowhere near “Acme Corp”.

Asking a colleague to write a script turns a two-hour task into a two-week dependency on someone else’s calendar, and produces something that cannot be adjusted when the next file arrives.

The three properties that decide the tool#

One column. This is the persona where the survey’s central limitation — that string-similarity tools cannot combine evidence across fields — costs nothing, because there is no second field. The tools this survey warns against elsewhere are correct here.

A person will confirm every merge. They know the domain. They can see that “Acme Corp” and “Acme Holdings” are different companies and that “Acme Corp” and “ACME Corporation” are not. Automation should propose; the person disposes.

It runs once. Reproducibility is worth something but not much. Nothing will be maintained.

The failure that matters#

It is not a wrong merge — the person will catch those. It is stopping too early.

Key-collision clustering catches variation in punctuation, case, word order and legal suffix, and it does so instantly. It does not catch a typo in the middle of a word, and it does not catch an abbreviation that changes the letters. Somebody who runs one clustering method, accepts the obvious groups and stops has usually fixed 70% of the problem and believes they have fixed all of it — with the remaining 30% now harder to find, because the obvious cases are gone and the survivors look like legitimately distinct values.

The counter is to run several methods in sequence and expect each to find fewer than the last, stopping when a method finds nothing rather than when the first one is exhausted.

What to reach for#

OpenRefine. It is the answer, and it is not close.

It is a desktop application driven through a browser, so no programming is required. Its clustering offers key collision (fingerprint, n-gram fingerprint, and phonetic keying via Metaphone3, Cologne, Daitch-Mokotoff and Beider-Morse) and nearest neighbour (Levenshtein and PPM) — several methods, which is exactly what the stopping-too-early failure needs. Each cluster comes with a suggested replacement the person edits or accepts, one at a time, which is the interaction this persona wants. It records an operation history, so if the file does arrive again the work can be replayed. And it is the healthiest project in this survey: 11,986 stars, 168 pull requests merged in six months, a push the day before this was written.

A spreadsheet’s own tools, for the trivial cases. Trimming whitespace and normalizing case before anything else removes a surprising share of the variants and takes one formula.

Not a Python library. Not for this person. The install alone — even the fastest here is 191 MiB and 20 packages — costs more than the task, and every one of them requires writing code.

And not a language model on its own. Pasting the column into a chat interface and asking it to group the values produces a plausible answer with no record of what was merged, no way to check the ones it got wrong, and no history if the file changes. As a way to suggest which variants to look at it is fine. As the thing that decides, it removes exactly the audit trail this persona’s one advantage — a person who knows the domain — depends on.

S4 approach#

The question#

Not “which is best” — S2 answered that for the general case — but which of these will still work in three years, and what happens to you when one of them stops.

This category has an unusually sharp answer available, because the survey window contained a live example. On 2026-08-19 a fresh pip install dedupe produced a working library. On 2026-08-20 the same command produced a library that installs, imports, and raises. Nothing about dedupe changed. BTrees released 6.5 and removed a method dedupe calls, under a dependency specifier that permitted it.

That is the failure mode this pass is about, and it is not the one people plan for.

What was examined#

The four cells of the stall test, per repository: open pull requests, pull requests merged in the last six months, open issues, and recent commit activity. Read as four cells rather than a ratio, because patches arriving with none merged and nothing arriving and nothing merged are different states with different futures, and this category contains clear examples of both.

The dependency ceiling. For every candidate: does it declare an upper bound on anything. A floor with no ceiling is a maintenance promise, and this pass is about what happens when the promise stops being kept.

What a project’s death actually costs its users. The variable is not project health — it is how much of your work is portable when the project stops. A library whose output is a cluster ID in your own table costs you a rewrite. A library whose model format only it can read costs you the model.

Governance and funding, read only as durability signals. What an organization raised answers “will this survive”, never “is this good”. Where a fact bears on a reader’s risk it appears with its source; nothing here ranks a vendor.

The three durability questions this category poses#

1. Is anyone merging patches? Measured directly. Four projects in this survey have inbound patches and zero merged in six months, and two of them are the category’s best-known names.

2. Is anything holding the dependencies still? The three runtime breakages measured in this survey — matplotlib’s get_cmap, torchtext’s legacy, BTrees’ byValue — all have the same shape. The library named a minimum version and no maximum, and something upstream removed an API. This is the dominant failure mode in the category and it is invisible from any health dashboard.

3. What survives if it stops? For each candidate: which artifacts are portable, which are trapped, and what the migration would actually involve. This is the question that decides how much a stall should worry you, and it varies more than project health does.

What is excluded, and why#

Vendor commercial position. Funding, valuation and market share are not evidence about software quality, and treating them as such is the 3.xxx arbiter role this series does not occupy. Pricing stays, because what a tool costs to use is a fact a reader needs.

Predictions about which project will be abandoned next. The stall test reports a present state. Three of the projects here that are stalled today were the category’s default recommendation at some point, which is a reason to describe mechanisms rather than to forecast.

dedupe — viability#

The four cells#

The stalled cell, unambiguously. Fourteen people wrote patches nobody has looked at, and the last push to the repository was over a year ago.

The cause is known and reflects badly on nobody#

Dedupe.io, the hosted service built on the library, shut down on 2023-01-31. The team returned their attention to consulting at DataMade. The library was always the open-source half of a commercial product; the commercial half ended, and the attention that the product paid for ended with it.

This is the most common way an open-source library in this category dies, and it should be separated from neglect. Nobody promised to maintain dedupe forever, it was given away freely for a decade, and 4,510 stars are a record of how good it was.

What is actually broken, and how bad it is#

A fresh install does not work. BTrees 6.5, released 2026-08-20, removed byValue(). dedupe calls it at canopy_index.py:76 under an unbounded BTrees>=4.1.4. Measured: AttributeError the first time it blocks. Full analysis in the-unbounded-dependency.md.

The fix is one line and nobody upstream will apply it. BTrees<6.5 in your own constraints makes the identical dedupe code run — every measurement in this survey used BTrees==6.4. So this is a severity problem for new adopters and a nuisance for existing ones.

It needs a C toolchain regardless. affinegap publishes no wheel past CPython 3.10 on any platform, and dedupe ships no aarch64 Linux wheel at any version. Both mean a compiler on the installing machine, which succeeds on a developer laptop and fails on a slim container image.

It ran out of memory on 5,000 records. MemoryError at a 6 GiB address-space cap, in prepare_training → coveredPairs, before any labeling. An uncapped run reached 7.9 GiB. Splink completed the same dataset in 2.24 seconds inside a 191 MiB environment.

What survives if it stops — and it already has#

The output is portable. Clusters with confidence scores, and canonical records. Once computed, they are yours and nothing about them depends on dedupe continuing.

The trained model is a settings file written by write_settings, readable by StaticDedupe. Portable in the weak sense that it survives a reinstall; not portable in the sense that anything else can read it.

The labels are the valuable artifact and they are portable. write_training emits JSON: the pairs a human judged and their verdicts. That is the expensive input in an entity-resolution project — a person’s hours — and it is not locked to dedupe. Those labeled pairs can train a logistic regression in recordlinkage, or seed a comparison in Splink, or simply serve as the evaluation set for whatever replaces it.

Anyone can fork it. MIT, 577 forks, and fourteen patches already written and waiting. The maintenance backlog is not hypothetical work; a substantial part of it is done and sitting in a queue.

The strategic reading#

Two different readers, two different answers, and conflating them is the error.

Already running dedupe in production: pin BTrees<6.5 today and change nothing else. Nothing has become wrong. The algorithm is sound, the canonicalization is unique in the category, and the environment you have works. Export the training data (write_training) and store it somewhere that is not the dedupe directory, because it is the part you cannot regenerate cheaply.

Choosing today: do not. The next breakage has the same shape as this one — an unbounded dependency, an upstream removal, an unattended inbox — and its date is set by somebody else’s release schedule. The category has a maintained alternative that scores comparably.

The exception is real, though. dedupe is the only candidate that canonicalizes, and its canonical record recovered a date of birth one input lacked. If producing a golden record is a hard requirement and the alternative is building canonicalization yourself, adopting a stalled MIT library with a pinned dependency and 577 forks behind it is a defensible trade. Make it as a decision, with the pin written down and the reason recorded, rather than arrived at by following a search result.

Re-check#

Whether anybody merges the fourteen patches, whether a fork gains traction, and whether affinegap ever ships a wheel past cp310. Any one of those would change the verdict.

S4 verdict — strategic paths#

What durability looks like in this category#

Not “is the project healthy”. Three of the projects here were the category’s default recommendation at some point and two of them now fail on a fresh install. The question that predicted those failures, and that this pass ends on, is narrower:

Is anyone merging patches, and is anything holding the dependencies still?

Both are one lookup. Together they explain every runtime breakage this survey measured, and neither appears on a health dashboard.

The three paths#

Conservative — the database, no dependency#

SQL in the engine you already run: blocking as a join condition, a similarity expression, a threshold, connected components. Measured at pairwise F1 0.815 and 73.1% of entities exactly right on FEBRL3, in 0.29 seconds and about thirty lines.

Durability: perfect. There is nothing to abandon. pg_trgm, fuzzystrmatch and DuckDB’s string functions are part of database distributions with decade-scale support horizons.

Ends when: you need term-frequency adjustment; when a fixed threshold stops absorbing better blocking — measured, the baseline was the only candidate made worse by improved blocking rules; or when somebody asks why two specific records merged and the answer has to be better than “the score was 0.83”.

Default — Splink on DuckDB#

One dependency, 191 MiB, 20 distributions, no labels, all four steps, and the best measured result in the survey: F1 0.977 and 97.7% of entities exactly right on FEBRL3 in 1.89 seconds.

Durability: the strongest available here, on two independent grounds. 210 pull requests merged in six months — the best ratio in the category, and the signal that fires first when a project stops. And the output is a cluster ID in your own table with the model as readable JSON, so a stall costs you a replacement library rather than your work.

The qualification: Splink declares dependency floors and no ceilings, exactly the shape that broke dedupe. What protects its users is that somebody is merging patches — a behavior, not a property of the metadata. That is worth knowing because behaviors change, and it is why the merge rate is the thing to re-check rather than the star count.

Ends when: entity IDs must stay stable across runs (see below), or the data outgrows a single machine — at which point the same model moves to its Spark backend, which is the migration this architecture was designed to make cheap.

Buy — a commercial engine#

Senzing, or a cloud service. Decided on three questions that need no evaluation: whether the data may leave, whether the decision must be explicable to an outsider, and whether the volume is steady or spiky.

Durability: a vendor relationship, with the usual properties. The specific fact a reader needs first is that evaluation itself is gated — the free license covers 500 records, below the 1,000 of the smallest dataset in this field’s own benchmark family, so a fair comparison against the open-source tier requires a sales conversation before any code runs.

Where it is genuinely the right answer: continuous resolution with stable entity identity, which the open-source tier serves worst. Every measured candidate is a batch process whose cluster IDs are reassigned on each run, and for a service whose IDs are foreign keys elsewhere that is disqualifying as shipped. Building the identity layer yourself is real, ongoing engineering — a stable ID store, a merge log, a policy for newly-linked clusters — and it is the part of the problem no library in this survey solves.

The decisions to make once, and write down#

Pin, and test unpinned on a schedule. Every breakage measured here reaches only a fresh install; a pinned environment is untouched by all three. A CI job that always restores from a lock file reports green through all of them, and the failure surfaces when a new developer joins or a base image is rebuilt. One weekly unpinned resolve-and-import would have caught the dedupe breakage on 2026-08-20 rather than on a Monday.

Store the labels separately if you ever produce any. A person’s hours judging pairs is the most expensive input in this category and the most portable artifact in it. dedupe’s write_training emits JSON that can seed a different library entirely. Keeping it outside the tool’s own directory costs nothing and survives the tool.

Export the benchmark data. If your evaluation depends on the FEBRL sets, they live inside recordlinkage, which has no maintainer. Exporting them with digests takes five minutes and removes an unmaintained package from your test path.

Prefer candidates whose output you own. Splink and recordlinkage both return ordinary objects — a cluster ID, a pandas index. Nothing in this survey traps its output in a format only it can read, which is unusually good for a category, and is worth preserving as a selection criterion rather than a coincidence.

Re-check list, with dates#

Decay class: fast. BTrees removed a method on 2026-08-20 and broke this category’s most-starred library eighteen days later. Splink released four days before this survey was written. In a category where the recommendation can be inverted by a dependency’s patch release, six months is the right window.

recordlinkage — viability#

The situation, stated plainly#

The most-installed dedicated entity-resolution library in Python has had no repository commit in over two and a half years, and no release in three.

The install number was checked hard, because it is the kind of figure that usually turns out to be an artifact:

• Not mirrors. 1,241 of 1,153,018 — 0.1%. dedupe’s mirror share, for comparison, is 41%.
• Not a spike. 181 days of daily data, running between roughly 15,000 and 47,000 a day.
• Not automated noise. The series dips on every Saturday and Sunday in the window. That weekday shape is continuous integration and office hours: organizations, repeatedly.

So over a million real installs a month, of a library nobody is maintaining.

Why so many, and what that implies#

Part of it is quality: it is the clearest explanation of how record linkage works that exists in Python, and its scikit-learn shape means anyone who knows that ecosystem reads the API in twenty minutes.

Part of it is the datasets. recordlinkage bundles FEBRL 1 through 4 and the German krebsregister cancer-registry set, with ground truth, inside the package. This survey installed it for exactly that reason. It is the easiest way to get labeled record linkage data in Python, and it is imported by coursework, papers and evaluation harnesses that never call the classifier. Some unknown share of that 1.15 million is people who wanted the data.

The implication for a reader is not that the install count is fake. It is that install count in this category is not a maintenance signal and should not be read as a recommendation — which is the same lesson the stars give, pointing at a different library.

The unusual position: stalled and in excellent shape#

Almost everything that kills a stalled library is absent here.

It installs in 1.7 seconds with no compiler, on both 3.12 and 3.13, as a pure-Python any wheel. Better install behavior than several maintained candidates in this survey, and far better than dedupe’s.

It still works, well. Its unsupervised ECM classifier produced results identical to Splink’s on FEBRL1 and within 0.7 points on FEBRL3, at 17× the speed. This is not a library limping; it is a library that was finished and left.

It declares a ceiling. pandas<3 is the only meaningful upper bound among the stalled candidates in this survey, and it is the reason recordlinkage has not suffered dedupe’s fate. Whether that was foresight or convention, it is the thing protecting its users.

Its dependency set is small and conservative: pandas, numpy, scipy, scikit-learn, jellyfish, joblib. Six well-maintained packages, none obscure, none compiled by a stopped maintainer.

The exposure has a name and a direction#

pandas 3.

Today pandas<3 protects recordlinkage. The day a reader’s environment requires pandas 3 for something else, the same pin stops being protection and becomes a constraint on everything else in the environment — and a library with no maintainer cannot lift it.

At that point the reader has three options, and all of them are cheap by the standards of this category: pin pandas, drop recordlinkage, or maintain a fork. The internals are pure Python over pandas idioms; the migration work is real but bounded and visible.

That is a materially better position than dedupe’s, where the equivalent event has already happened, arrived without warning, and left a compiled dependency chain behind it.

What survives if it stops — it already has#

Everything, essentially. The output is a pandas MultiIndex of pairs and a DataFrame of comparison vectors. Both are ordinary pandas objects. There is no model format, no serialized artifact, nothing that only recordlinkage can read.

The model is a published one. ECM is Fellegi-Sunter with expectation-maximization — 1969 statistics, implemented independently in Splink and in R’s fastLink. A team moving off recordlinkage is changing libraries, not changing methods, and this survey measured the two Python implementations agreeing exactly on one dataset.

The datasets are the exception, and they are the thing worth extracting. If the package ever stops installing, the FEBRL sets go with it for anyone who has not exported them. This survey’s harness exports them to CSV with SHA-256 digests for that reason, and that is a five-minute precaution any team relying on them should take.

Verdict#

Safe to keep. Not the thing to build on.

For a pipeline already using it, pinned and working: no urgency at all, and a clear trigger to watch — the day pandas 3 becomes a requirement elsewhere in the environment.

For learning how the category works, and for research comparing classifiers on the same comparison vectors: still the best thing in Python, and its being unmaintained barely matters for that use.

For something new that has to run in 2030: the release date is the fact. Splink implements the same model, produced the same answer, is being maintained, and clusters as well.

Re-check#

Whether anyone picks it up, and how quickly pandas 3 becomes unavoidable. The second is the clock, and it is not one the project controls.

Splink — viability#

The healthy signals, and what they are worth#

210 pull requests merged in six months against 25 open is the best ratio in this survey by a wide margin, and it is the signal that fires earliest when a project stops. It says the inbox is attended, which is what the three broken libraries in the-unbounded-dependency.md lack.

The release cadence supports it: 163 releases since the project began, one four days before this was written.

Institutional backing is the durability argument, and it cuts both ways#

Splink is maintained by a government department that uses it in production to link criminal justice records, and has to explain its methodology in public. Three consequences follow.

It does not depend on a startup’s runway. The commonest death in open-source data tooling is a company pivoting or being acquired, and there is no company here. Splink was built because the department needed it and published because publishing was policy.

Its diagnostics are unusually good, for a structural reason. Match weight waterfalls, parameter estimate charts and a comparison viewer exist because the people who built them get asked to justify individual decisions. Very little else in this category can show its working, and that is the artifact the public-interest persona depends on.

Its roadmap answers to a public-sector data problem. The model of the world is people, addresses and dates. Nothing prevents its use on products or companies, and people do it — but the comparison library and the term-frequency handling are shaped around the first case, and a reader working on object catalogs or supplier names is using a tool aimed slightly past them.

The risk specific to institutional ownership is that a department reorganizes, a team moves, and a project with no commercial pressure to continue simply stops shipping. It has happened to government-published tools before. What limits the damage here is the second half of the durability question.

What survives if it stops#

More than for anything else in this survey, and this is the strongest single argument for it.

The output is a cluster ID in your own table. Not a proprietary format, not a model file only Splink reads. If Splink stopped tomorrow, the entity assignments already computed are unaffected and remain yours.

The model is a JSON settings object. Comparison definitions, blocking rules and estimated m and u probabilities, readable and portable. Somebody could reimplement scoring against those parameters — the Fellegi-Sunter arithmetic is a published 1969 model, not a secret — and recordlinkage’s ECMClassifier already implements the same thing.

The generated artifact is SQL. The comparison logic leaves the library as a query. A team that needed to could capture the generated SQL and run it without Splink at all.

There is a live alternative implementing the identical model. recordlinkage’s ECM produced identical results on FEBRL1 and came within 0.7 points on FEBRL3. That it is itself unmaintained is a real qualification — but “the model is implemented twice in Python” is a materially different position from “one library, one implementation, no specification”, which is where several of this survey’s dead candidates were.

The exposures#

No dependency ceilings. Splink declares floors on duckdb, sqlglot, pandas and numpy and no maxima — the same shape that broke dedupe. The difference is entirely that somebody is merging patches, which converts the exposure from a time bomb into a maintenance task. So Splink’s safety here is a behavior, not a property of its metadata, and behaviors change.

sqlglot is load-bearing and invisible. All SQL generation goes through it. A dialect regression in a transitive dependency surfaces as a backend-specific Splink bug, and the number of people who can diagnose that is small.

Version 4 broke version 3. It happened once, it was documented, and the search-engine sediment is still mostly Splink 3. A newcomer’s first hour is worse than it should be, and a second major version would do the same again.

The DuckDB coupling is real. DuckDB is a hard dependency, not an extra, even for users who intend to run on Spark.

Verdict#

Adopt. The best-maintained candidate in the category, the best measured result, the most portable output, and a second implementation of its model already exists. The realistic bad outcome is not that it breaks — it is that the team moves on and it slowly stops following pandas, at which point everything already computed is still yours and the model parameters are still readable.

Re-check in six months: whether the merge rate holds, and whether a Splink 5 is signalled.

The unbounded dependency#

This category’s dominant failure mode, with three worked examples measured in one sitting, and a date attached to the most recent.

The pattern#

A library declares a minimum version of a dependency and no maximum:

Requires-Dist: BTrees>=4.1.4

While the library is maintained, that is correct and generous. It means the library intends to follow its dependencies, and pinning a ceiling would force downstream users into resolver conflicts for no benefit.

The day the maintainer stops, the same line inverts. It no longer says “we will keep up”. It says “we will accept any future version of this dependency, sight unseen, forever” — and the first upstream release that removes an API the library calls turns a working install into a broken one, on a date the library’s users did not choose and nobody announced.

Three examples, all measured on 2026-09-07#

BTrees 6.5 → dedupe. Released 2026-08-20, eighteen days before this survey. Its changelog: “Remove the long deprecated byValue() method from all Bucket and BTree classes except those of the II and LL families”, with the cause given as a C-level crash — a value whose comparison method mutated the bucket freed the array the C code was iterating over, issue 226.

dedupe 3.0.3 calls results.byValue(...) at canopy_index.py:76, on an IFBucket, which is in the removed set. It pins BTrees>=4.1.4. Measured:

AttributeError: 'BTrees.IFBTree.IFBucket' object has no attribute 'byValue'

raised the first time it blocks. Bisected across 5.2, 6.0, 6.1, 6.2, 6.3 and 6.4 — all have the method; only 6.5 does not. BTrees==6.4 makes the identical dedupe code run.

Matplotlib 3.9 → polyfuzz. get_cmap was removed from matplotlib.cm. PolyFuzz 0.4.3 imports it at module scope, so the failure is at import polyfuzz, before any user code executes. 802 stars, 36,338 installs a month, four open pull requests, none merged since 2026-03-01.

torchtext → deepmatcher and entity-embed. torchtext.legacy was removed in 2022; deepmatcher imports it. torchtext then stopped shipping wheels after CPython 3.11, so entity-embed cannot even resolve. Both are still listed in the community’s index of this category alongside Splink.

Why this category is exposed more than most#

The libraries are small and the dependencies are large. dedupe is a matching algorithm that leans on BTrees, zope.index, scikit-learn and six compiled micro-packages. recordlinkage is a pipeline over pandas, numpy, scipy and scikit-learn. The surface area of other people’s APIs is much larger than the library’s own, and every one of those is a place a removal can land.

The dependencies are load-bearing and old. dedupe’s chain includes highered (last released 2016), simplecosine (2017) and categorical-distance (2016). Those are stable because they are finished, which is fine — but they are also the reason the chain has more members than a reader would guess, and each member is another maintainer who may or may not still be there.

Maintenance in this category is thin. Four projects here have inbound patches and zero merged in six months. A removal upstream that would be fixed in a day on a healthy project is permanent on a stalled one.

The signal that predicts it, and it is not project health#

A dashboard showing stars, downloads and last-release date reports all three of the broken libraries above as fine or nearly fine. polyfuzz released in May 2025 and does 36,338 installs a month. dedupe has 4,510 stars.

The two facts that actually predict the failure are:

1. No upper bound on a dependency that is still moving. Check the Requires-Dist lines. A ceiling — even a loose one — means somebody thought about it.
2. Nothing merged recently. Not “no commits”, which can mean finished, but specifically patches arriving and none merged, which means the inbox is unattended.

Together they say: this library will break, the date is set by somebody else’s release schedule, and no one will fix it.

What follows for a reader#

The lock file is the mitigation, and in this category it is not optional. Every breakage above reaches only a fresh install. A pinned, working environment is untouched by all three. In a category where four of the best-known projects have unattended inboxes, “we pin everything” stops being hygiene and becomes the load-bearing part of the strategy.

Test a fresh install on a schedule, alongside the pinned one. A CI job that always restores from a lock file will report green through every one of these events. The failure surfaces the day a new developer joins, or a base image is rebuilt, and by then it is urgent. One weekly unpinned resolve-and-import job would have caught the dedupe breakage on 2026-08-20 instead of on a Monday morning.

Prefer a candidate whose output you own. This is the strategic version of the same point. If the artifact is a cluster ID in your own table, a breakage costs you a replacement library. If the artifact is a model file only that library reads, it costs you the model as well.

And read the ceiling before adopting. It takes one look at the metadata and it is the single most predictive thing available about a small library’s next three years.