Is it possible to have a process for developing new DSLs? In this article I’m going to describe my rather informal personal process, and I’d be interested to learn how you approach DSL development.
My process starts with talking to the customer about the new feature they want built. I’ll ask probing questions to figure out if maybe we shouldn’t be doing the feature just yet. Perhaps a library that we use, such as mbeddr.platform or iets3.opensource (aka KernelF), provides the feature already. Maybe we can combine two other features to achieve what we want?
Often though we do have to build a new feature. I keep calling it “feature” here but usually in DSL development a new feature takes the form of one or more concepts to add. The next step in my personal approach is to take a piece of paper and think about these new concepts. The goal is to figure out what the “rules of the game” should be: what should I be allowed to express, what should not be allowed and should be checked against. I try to think of existing features of general programming languages or other DSLs that could be similar. Are we trying to implement a data type definition? Or is it like a business rule? Maybe it’s similar to a tree of calculations? A query? A plugin mechanism for extensibility? All these ideas have a “precedent”, existing concrete implementations.
The next step is to either implement the idea if it is simple or describe it and gather feedback from the customer. At this point a complicated idea that exists in my head will have to be “serialized” and converted to a stream of words. If I find it difficult to describe the proposed design at this point, perhaps it’s something that I should change and simplify before moving forward with it. On the other hand, if I found a “precedent” in the previous step, that usually makes the idea that much simpler to explain.
Now it’s time to implement the feature. Sometimes it consists of many rather stereotypical concepts: I have to implement a smart reference, a new expression or an “activity action” (a term from a client project). It helps to keep a todo list of items to implement, and I also experimented with keeping a checklist of things to remember when creating one of those stereotypical concepts. A checklist or a todo list is additional bureaucracy and may sound limiting but I found that past a certain threshold it gives me freedom to implement the technicalities in any order I want, knowing that I will get to everything eventually and won’t have to revisit the code again later after the customer finds out that I forgot to specify the scope of a reference, for example, or create a migration. I usually keep my todo list in VS Code in Markdown, same with checklists.
After the feature is implemented, the next step is to gather feedback. Often in my current project I can just push the code and tell the customer that I’m done and they will check out the code and test out the feature. Sometimes I like to take a screenshot or even an occasional screen recording. I use Loom to make these short videos where I record both the screen and myself talking in a little circle in a corner.
At this point we’re either done or the process loops back to one of the previous stages.
If you develop DSLs for a living, does your process look similar to mine? How is it different? Let me know.