Continuing on my deduplication adventures that I wrote about previously. It turns out, the client does not need any structural similarity checks, and it is enough to just check the names for duplicates.
As I wrote previously, I was unable to get the agent to look for duplicates without tooling. However, I must admit I didn’t try too hard either. I probably could have altered the prompt or disabled all the tools but the problem is, I wouldn’t be able to trust the reply too much. What if the model overlooks a clear duplicate? What if it hallucinates one that is not there? Well, this last problem can be solved by asking the model for a proof and checking it by a deterministic script. But as soon as deterministic scripts enter the picture, why not do more work in such a script and only use the AI for the parts that cannot be automated?
Thinking about the problem some more, I decided to ask the model to write a “canonicalizing” script: I give it a list of German/English names such as ‘DSGVO’, ‘Datenschutz-Grundverordnung’, or ‘GDPR’, and out comes “general data protection regulation”. This turns out to be a very simple deterministic script, all the difficulty lies in the data: you have to have a German-English dictionary and a glossary of abbreviations.
So the hard problem is now reduced to providing this dictionary. And this is a task that should nowadays be solvable even by a rather simple language model: “Given the task of searching for duplicate names, this list of English/German names, and this current dictionary/glossary, what entries are missing in the dictionary?”
Overall, the reason for using a deterministic script rather than an LLM seems to be analogous to the reason to use a machine instead of a human: machines are faster and more reliable than humans, but need some initial investment and ongoing maintenance by humans. Deterministic scripts are faster and more reliable than LLMs but need some initial investment and maintenance (by humans, but perhaps even LLMs would be enough?)
Anyway, the next step would be to set this up as a simple workflow/check, including regular dictionary updates and some machinery to mark duplicates as acceptable.