Findings this week:

  1. Agentic code review works for MPS libraries, as evidenced by the recently merged PRs to MPS-extensions: #1932, #1933, #1940, #1942, #1943. Most of them contain a dialogue between my agent (initially as myself, later as @spclngs-bot) and Eugen Schindler’s agent.

  2. Agentic code review works especially well when the implementor is also an agent. It surprised me that even an “unskilled” agent can produce code that is up to standards when the review guides it. PR #1933 started with the agent writing code to access MPS models in plain Java, looking up concepts and properties by name, rather than using the idiomatic smodel language. The reviewer reported this, the agent made the necessary changes, and the code looked fine in the end.

  3. Agentic code review is only as good as the standards it is reviewing against. If your standards don’t mention anything about higher-level DSLs, then plain Java code would pass the review.

  4. In case you’re curious, one review of one PR has cost about $2-$5 in tokens, as estimated by Claude. Each PR needed roughly 4-5 rounds of reviews before it was good enough to be merged.

  5. To fight slop, it helps to have pre-commit checks that keep things small: limit files to 20000 bytes, limit Markdown paragraphs to 150 words. This is something I have been experimenting with locally, on my internal repository that contains the review automation scripts. I haven’t tried a rule that would limit MPS root nodes to 1000 nodes, or something to that effect, on MPS-extensions, but this is an idea.

  6. You can (and should) come up with new guidelines when looking at the existing code, but retrofitting them on the existing code remains painful even with agents.

  7. If you decide that time has come to automate the PR reviews using an agent, beware that you then have to maintain the automation, and it may become a distraction in itself. This is also the reason why my today’s email is so brief: I’d rather write programs to write programs than write programs (or emails).