A client of mine is modeling components in MPS. They are a large, multinational manufacturing company, so they have hundreds to thousands of them. Some are fully fleshed out, some are just stubs that have nothing but a name. The modeling task is split across several teams, and two teams may end up modeling the same component, for example, one that they need as a dependency.
How do we detect these duplicates? It sounds like a fairly simple problem, just collect them and check for similar names. It is, however, made complicated by the fact that one team may choose to model the components in English while another models them in German. Or one team uses an abbreviation and the other spells out the full name. Language translation is something that large language models have always been good for, so it feels obvious that AI could be part of the solution. But what should be its place exactly?
I decided to start simple: let’s export all components to a text file, feed the file to the LLM and ask it if it sees any duplicates. In fact, even simpler: let’s just export the component names, paste them in, and ask the LLM to canonicalize them by translating from German to English and expanding abbreviations where possible.
Sounds trivial? Well, not so fast. One does not simply paste a 3,000-line file into a Copilot prompt. Copilot just stows it away in some session-specific directory and tells the agent where it is. This is basically the same as @-mentioning it in the prompt. The result is that the LLM only sees the file name and instead of trying to produce an answer from context, it reads a few hundred lines to understand what it’s up against, then rolls up its sleeves and starts writing Python scripts. Just like a real, lazy programmer: it’s far more interesting to spend a day writing a script that takes a minute to run, than producing an answer in five minutes. (Of course, being a lazy programmer myself, I kind of empathize and agree with the approach. Especially since my goal is to automate this and run it repeatedly, so I welcome deterministic scripts.)
Interestingly, in the scripts it writes, it will include a small German-English dictionary, based on the actual words it encounters in the input. So this gave me another idea: structure the workflow so that we do as much mechanical work as possible outside of the LLM, and only give the LLM a simple, focused translation task. This kind of goes against The Bitter Lesson (which says that general-purpose methods will outperform special-purpose human-engineered approaches given enough compute), but I guess that lesson applies only as a macro trend, not on a micro scale.
All this was just dealing with the names. Who knows, maybe in the end just using some structural similarity score will fare better. We’ll see next time.