No single tool deduplicates and normalizes an entire stock library in one pass, which is why this question keeps getting confusing answers. The job splits in two. A metadata engine such as ExifTool, Xpiks or Lightroom Classic finds and removes duplicates inside each file. A controlled vocabulary decides which form of each word is allowed to survive. Most contributors only ever do the first half.
That second half is the difference between a library that is merely tidy and one that is consistent. What follows is what each tool actually does, where it stops, and how to run the operation across thousands of files without damaging a portfolio that is already earning.
What counts as a duplicate keyword
Contributors usually mean one of five different things when they say duplicate, and the tools handle them very differently.
- Exact repeats. The same string listed twice in the same field, usually the result of appending a preset on top of metadata that already contained it.
- Case variants. Sunset and sunset. Harmless in some agency search indexes, counted as separate terms by others.
- Singular and plural. Tooth and teeth, tree and trees. Xpiks calls these semantic duplicates and warns about them explicitly, noting that some microstocks treat the pairing as spam.
- Synonyms. Car, automobile, vehicle. Not duplicates in a strict sense, but three slots spent on one concept when the keyword budget is finite.
- Cross-field repeats. The same terms sitting in the title, the description and the keyword list at once. Some reviewers read that as stuffing.
A pass that only catches the first two is the common outcome, and it is also the least useful one. The real gains sit in the third and fourth categories.
What each tool actually does
Xpiks
- Best for: seeing duplicate and spelling problems while you edit, a folder at a time.
- How it works: with duplicate detection enabled, matching terms are highlighted in the keywords, title and description fields, and a Show duplicates view isolates them for fixing.
- Pricing model: a free desktop version with a paid Pro tier. Batch keyword deletion, find and replace, and CSV import sit in the paid tier.
- Main limitation: it is a per-session desktop tool. There is no persistent vocabulary governing every future file, so you are fixing folders rather than enforcing a standard.
ExifTool
- Best for: technical users who want a scriptable, repeatable pass over tens of thousands of files.
- How it works: each keyword is removed and re-added within the same command, which collapses repeats instead of stacking them. Wrapped in a shell loop, it walks an entire archive unattended.
- Pricing model: free and open source.
- Main limitation: it does exactly what you tell it and nothing else. No opinion about synonyms, no agency awareness, and no undo beyond the backup copies it leaves behind.
Lightroom Classic and Adobe Bridge
- Best for: photographers whose catalogue is already the source of truth for their images.
- How it works: a central keyword list with hierarchy and synonyms means each term is defined once, and renaming it in the list propagates to everything it was applied to.
- Pricing model: subscription, bundled into the wider Adobe photography plan.
- Main limitation: the vocabulary lives inside the catalogue. Files that were never imported, or that arrived from another machine, sit outside it entirely.
Spreadsheets
- Best for: smaller libraries, and for anyone who wants to see the whole vocabulary in one view before deciding anything.
- How it works: export metadata to CSV, deduplicate and normalize with formulas or a pivot table, then re-import.
- Pricing model: free, assuming you already have a spreadsheet application.
- Main limitation: the round trip is where portfolios get damaged. One mismatched filename column and metadata lands on the wrong images.
Rastock AI
- Best for: contributors who want the vocabulary enforced at generation time rather than repaired afterwards, and who deliver to several agencies at once.
- How it works: metadata is generated from the image, then checked against per-destination rules. Banned terms are removed, mandatory terms are present, duplicates are collapsed before delivery, and a batch that comes out wrong can be rolled back instead of corrected file by file.
- Pricing model: subscription, with a 14-day free trial and no card required. No revenue share and no ownership claim over your files.
- Main limitation: it governs what goes out from now on. Files already live in an agency portfolio still have to be re-uploaded or edited on the agency's own side, and that constraint applies to every tool on this list.
When to use which
If you have one messy folder and want to understand the problem, use Xpiks. The visual highlighting is the fastest way to see what is actually wrong with a keyword set before you commit to a rule about it.
If you have twenty thousand files and a rule you already trust, use ExifTool. Scripting is the only approach that reaches that count without a subscription, and the operation is deterministic, which matters when you cannot inspect every result.
If your catalogue is the centre of your workflow, use Lightroom's keyword list. A hierarchy defined once beats a cleanup pass repeated every quarter, and the synonym handling does the normalization work for you.
If the problem is that new files keep arriving with the same inconsistencies, none of the above addresses the cause. That is a generation-side problem, and it needs a tool that applies the vocabulary before the file is delivered.
The normalization step people skip
Deduplication removes repeats. Normalization decides which form is correct, and it is the half that actually changes search performance.
A workable vocabulary answers four questions and then never changes its mind: singular or plural for each concept, British or American spelling, which synonym in each cluster is canonical, and whether compound concepts are recorded as one keyword or two.
Write those decisions down before touching a single file, and settle how the keyword set is structured at the same time. A deduplication run without a written rule just produces a differently inconsistent library.
It also helps to know which terms are worth keeping. Keyword performance data from the agencies tells you which of your competing synonyms actually pull downloads, which turns an arbitrary choice into an evidence-based one.
Running it without breaking a live portfolio
- Copy first. Run the operation on a duplicate of the archive, never on the archive.
- Start with a hundred files rather than ten thousand, and inspect the output in a metadata viewer before scaling up.
- Check both IPTC and XMP. Keywords can live in both, and a tool that writes one while leaving the other stale replaces your old inconsistency with a new one.
- Keep the originals until any re-upload has passed review. Backups are only useful before you need them.
- Do it in one pass per file. Repeated partial edits are how most of these duplicates got there in the first place.
If the library is genuinely old
There is a real difference between normalizing a working library and rescuing keyword sets written years ago under assumptions nobody remembers. The second is a heavier job with different priorities, and it is covered separately.
Where this leaves you
Deduplication is a one-time repair. Normalization is a standing rule. The tools listed here handle the repair well and the rule badly, which is the honest reason libraries drift back into inconsistency within a year of being cleaned.
Rastock.ai was built for the rule side of that split: metadata generated once, checked against each agency's requirements, and delivered to 10+ agencies over FTP or SFTP, with IPTC and CSV export left open so nothing is trapped inside it. The plans and the free trial are listed openly. For a library that is already inconsistent, though, one of the repair tools above still has to run first — no delivery tool retroactively fixes what is already published.
Frequently asked questions
Can I remove duplicate keywords from thousands of photos at once?
Yes, but not with a single click in most tools. ExifTool will do it across an entire archive from the command line by removing and re-adding each keyword in the same operation. Xpiks surfaces duplicates visually across a folder at a time. Spreadsheet round trips work for smaller libraries. Whichever route you take, run it on a copy first and inspect a sample before scaling up.
Are singular and plural forms of a keyword treated as duplicates?
Some agencies treat them that way. Xpiks explicitly flags singular and plural pairs as semantic duplicates and notes that certain microstocks read them as spam. Neither form is universally correct, so the practical answer is to pick one form per concept, write the decision down, and apply it consistently rather than leaving both in place.
Will deduplicating keywords change my rankings on stock agencies?
It can move either way, and nobody can promise a direction. Removing genuine repeats and stuffing tends to help, because reviewers and search indexes penalise both. Removing a synonym that was quietly bringing in downloads will cost you. This is why keyword performance data matters before the cleanup rather than after it.
Should keywords be normalized in the file or in a spreadsheet?
In the file, if you can. Metadata embedded as IPTC and XMP travels with the image and cannot drift out of sync with a separate document. A spreadsheet is easier to read and edit in bulk, but the export and re-import round trip is where portfolios get damaged, usually through a mismatched filename column.
Does removing duplicates require re-uploading files to agencies?
For files already live in an agency portfolio, yes in most cases, or you edit the metadata through the agency's own contributor interface. Cleaning the local copies changes nothing on the agency side by itself. This constraint applies to every tool available, which is why enforcing a vocabulary on new files is cheaper than repairing old ones.
What is the safest order of operations?
Decide the vocabulary rules first and write them down. Copy the archive. Run the deduplication on a hundred files and inspect the result in a metadata viewer. Scale up only once the sample is correct. Keep the originals until any re-upload has passed review. Reversing this order is how libraries end up worse than they started.