The Problem
When a company acquires other companies, it inherits their ERP systems. Each system has its own customer table with its own naming conventions, address formats, and identifiers. The same real-world customer appears in multiple systems under different names. Boeing might be "Boeing Co." in JD Edwards, "THE BOEING COMPANY" in GEAC, and "Boeing Defence UK Ltd" in tcmIS.
For this project, I simulated exactly this scenario: five separate ERP systems, each with their own customer records, schemas, and data quality issues. The goal was to build a Master Data Management (MDM) layer that could take 333 source records and resolve them into golden customer entities, with full cross-reference traceability.
Why Splink
The traditional approach to customer matching is deterministic: exact match on tax ID, or a series of business rules comparing name, city, and state. This works when your data is clean. It fails when "Boeing Co." and "THE BOEING COMPANY" need to match, or when addresses are formatted differently across systems.
Splink is an open-source Python library from the UK Ministry of Justice that implements the Fellegi-Sunter model for probabilistic record linkage. Instead of hard-coding rules, you define comparison columns and let the model calculate match probabilities using techniques like Jaro-Winkler string similarity. Each potential pair gets a match weight. You set a threshold, and pairs above it become linked entities.
I chose Splink over alternatives like RecordLinkage or dedupe because it scales well on Spark (important for the Databricks version), has excellent documentation, and produces transparent match weights that are auditable. In MDM, auditability matters: you need to explain why two records were linked.
The Pipeline
The architecture follows the same medallion pattern as the rest of the platform:
Bronze: Node.js extractors pull raw customer records from each of the five simulated ERPs into PostgreSQL. Each record retains its source_system identifier and original primary key.
Silver:dbt Core models normalize the records: standardize name casing, strip punctuation, parse addresses into components, and create a clean name_clean column for comparison. This is where "THE BOEING COMPANY" becomes "boeing company" and "Boeing Co." becomes "boeing co".
Splink Matching: A Python script runs Splink against the normalized Silver records. Comparison columns include name_clean (Jaro-Winkler), city, state_province, postal_code, and tax_id (exact match where available). The model produces cluster IDs grouping records that refer to the same entity.
Golden Records:A final step selects the "best" record from each cluster as the golden record, using a priority ranking by source system. The cross-reference table maps every source record to its golden ID, enabling consolidated sales queries across all five ERPs.
Results
From 333 source customer records across five ERPs, Splink identified 82 duplicate clusters and produced 251 golden customer entities. The cross-reference table provides full traceability: for any golden record, you can see exactly which source records contributed and from which ERP system.
The consolidated sales view joins order data from all five ERPs through the golden customer ID, giving a single revenue picture per real-world customer. This is the foundation that any quoting, pricing, or credit system would need to sit on.
Databricks Serverless Version
I later rebuilt this on Databricks Serverless with a DuckDB backend. DuckDB is compatible with Serverless compute (which does not support Spark), so Splink runs in single-node mode with DuckDB as the execution engine. The synthetic dataset uses the same five-company aerospace scenario. This version demonstrates that probabilistic record linkage works on Databricks even without a Spark cluster.
What I Learned
The hardest part of MDM is not the matching algorithm. It is the normalization. If your Silver layer does not standardize names, addresses, and identifiers consistently, even the best probabilistic model will miss matches or produce false positives. I spent more time on the dbt normalization models than on the Splink configuration.
The second lesson: match weights need to be visible. Stakeholders want to know why "Boeing Co." and "THE BOEING COMPANY" were linked. Splink produces a match probability for every pair, broken down by comparison column. That transparency is what separates a trustworthy golden record from a black-box dedup.