Find and merge duplicate values
Heuristicmatching_deduplicateBest-effort — fuzzy, approximate, or inferred; check the confidence score.
Cluster a list of strings by fuzzy similarity to find likely duplicates. Returns groups with a canonical value and its duplicates with scores. Threshold controls the minimum similarity for clustering, 0.0-1.0 (default 0.85). Useful for data cleaning and dedup. Tip: for organization or facility names, run names_org_name_clean on each item first — normalizing casing and punctuation lifts the similarity scores. This is raw fuzzy string matching, not semantic, so heavy abbreviations ('Univ' vs 'University', 'SLU') may still not cluster; lower the threshold or pre-expand abbreviations for those.
Parameters
itemsarrayrequiredList of strings to deduplicate.
strategystringoptionalSimilarity strategy: ratio (character-level), partial_ratio (best substring), token_sort_ratio (ignores word order), token_set_ratio (ignores order + duplicates, default).
thresholdnumberoptionalSimilarity threshold for clustering (0.0–1.0).
Call it
curl -X POST https://api.snipget.ai/v1/matching/deduplicate \
-H "Authorization: Bearer sk_live_..." \
-H "Content-Type: application/json" \
-d '{"items":["Acme Corp","ACME CORP","Acme Corporation","Zebra Inc","zebra inc."],"strategy":"token_set_ratio","threshold":0.85}'Example response
{
"clusters": [
{
"canonical": "Acme Corp",
"index": 0,
"duplicates": [
{
"value": "ACME CORP",
"index": 1,
"score": 1
}
]
},
{
"canonical": "Zebra Inc",
"index": 3,
"duplicates": [
{
"value": "zebra inc.",
"index": 4,
"score": 0.9474
}
]
}
],
"cluster_count": 2,
"unique_count": 3,
"duplicate_count": 2,
"strategy": "token_set_ratio"
}FAQ
What does this do for me?
Give it a list of values and it groups the ones that look like the same thing, so you can spot and merge duplicates like 'Acme Corp' and 'ACME CORP.' It is built for cleaning up name and label columns where the same entity was typed slightly differently.
Will it catch every duplicate?
No. This is a heuristic, best-effort grouping based on text similarity, so it can miss real duplicates that look very different and occasionally group two values that are not actually the same. Each grouped item carries a score, and you should review the clusters before deleting or merging anything.
How do I make it stricter or looser?
Use threshold (0.0 to 1.0, default 0.85). A higher threshold only groups very-close matches and produces fewer, tighter clusters; a lower one groups more aggressively and risks false merges. Start at the default and adjust if you see too many or too few groups.
What do the numbers in the response mean?
Only groups that actually contain duplicates are returned. cluster_count is how many such groups were found, duplicate_count is the total extra copies across them, and unique_count is how many distinct items remain after grouping. Each group has a canonical value and its matched duplicates with scores.
Does it pick the 'right' name to keep as canonical?
No. The canonical is simply the first item in each group by list order, not a judgment about which spelling is best. If you care which version survives, sort your list so the preferred form comes first, or pick the keeper yourself from the returned group.