Author disambiguation at scale

Bibliometric databases store author name strings, not identities. The same researcher appears under different name formats across their papers; different researchers share the same name. Ask “which papers did this person write” and a naive matcher either splits one career into fragments or merges strangers into one profile.

The merge direction is the dangerous one. A common name can snowball — each bad merge makes the next one look more plausible — until a false mega-cluster absorbs researchers who share nothing but a surname. A system at this scale needs safeguards against that failure mode, not just a good similarity score.

The pipeline is a two-stage Python ETL. PostgreSQL 15 with pgvector holds the records and their 384-dim embeddings; Elasticsearch with a Turkish analyzer covers fuzzy name search; SentenceTransformers produces the embeddings and RapidFuzz handles string similarity.

Matching runs through a hierarchy of signals rather than one blended score: email, ORCID, ResearcherID, affiliation, the co-authorship graph, and finally semantic embedding similarity, in that order of trust. The hierarchy can also veto — two records carrying different ORCIDs are rejected outright, no matter how well every other signal agrees. A level that stops a merge even when everything else says yes is the most useful decision in the system. On top of the pairwise decisions, anti-snowball safeguards protect against false mega-clusters.

Scale was the other half of the work. A MinHash LSH block-then-cluster candidate layer removed the Elasticsearch bottleneck entirely, and an out-of-core memmap design keeps 85M-scale processing inside a fixed memory budget. The single largest runtime win was mundane: replacing an O(N²) read pattern with O(N) keyset pagination.

A ThreadPool Elasticsearch prefetch was supposed to overlap I/O with compute. Measured end to end, it made the pipeline 35% slower. And a bucket filter that looked like a cheap optimization cost 30 cases on the ground-truth accuracy benchmark.

Both changes read as improvements and died on measurement — which is the argument for keeping a ground-truth benchmark next to the pipeline.

~92% accuracy on a 200-case ground-truth benchmark, with cluster-quality guards.

Full-pipeline runtime on a 2.5M-record dataset roughly halved: ~4h → ~2h.