1. The input file can be unparseable as JSON. For example, the user has chosen a .zip file by mistake instead of .json.
2. A valid JSON file can have unexpected structure. For example, the user has chosen a .json file with a different, unrelated data set.
3. The data set is correct but contains incorrect values. For example, a reference mentions an object that does not exist, a field value is out of expected range, or some references form an unexpected cycle.
4. The file could be fully valid but unexpectedly large.

We often want to avoid leaving the model in an inconsistent state after a failed import. While we can sometimes assume that users are using version control and know how to roll back unintended changes, it is nicer if they don’t have to. Our current importer logic adds the objects straight into the target model as we go through the parsed structure, and as a result it is prone to this issue.

We can fix this by importing objects into a temporary model first, and only moving them to the target model once the full import is complete. While in theory the move could partially fail as well, in practice this is unlikely as it all happens in memory with data that has passed validation.

To create a temporary model we can make use of jetbrains.mps.smodel.tempmodel.TemporaryModels singleton:

TemporaryModels temporaryModels = TemporaryModels.getInstance();
model tempModel = temporaryModels.createReadOnly(TempModuleOptions.forDefaultModule());
try {
  importDataIntoModel(sourceFile, tempModel);

  foreach root in tempModel.roots(<all>) {
    targetModel.add root(root);
  }
} finally {
  temporaryModels.dispose(tempModel);
}

I have moved the original code into importDataIntoModel method that we now call to import the data into the temporary model. After the import is successful, we add all roots to the target model. Adding a root to another model removes it from its original model.

When creating a temporary model we have to choose whether it will be read-only or editable. This pertains to the ability of the user to edit it via an editor, it is possible to modify the model via code in any case. We do not show the model to the user and in any case we don’t intend it to be modifiable by the user, so we choose the read-only option.

In the next post we will make it possible to update already existing data via import, rather than adding new objects on each import. In the meantime, the code is on GitHub.

We can’t!

We cannot change the concept of a node directly. There are no API methods for it, and in fact it is stored in a final field in the SNode class.

Here is what we can do instead:

1. Replace the node with another node that has the same ID.
2. Move properties, references and children from the original node to the new node as needed.
3. Fix any direct references that we know about.

A direct reference is a reference that internally stores a Java object reference to the target node. An indirect reference instead stores only the identifiers of the target node and its model and will look up the target node object when needed.

Any direct references that we don’t know about in step 3 will get broken, but it is usually not a problem because direct references are used only in a few specific scenarios:

• In the generator, if a reference macro returns a string rather than a node.
• When a node is detached (deleted), all references in the detached subtree will be made direct.

So as long as we a) do not perform these tricks in the generator (and we most probably don’t need to), and b) fix the reference targets of all the new descendants added in step 2, we should be safe.

The easy way

MPS provides a method that performs the three steps above, RefactoringRuntime#replaceWithNewConcept(originalNode, newConcept) (MPS link).

This method assumes that the existing node is fully compatible with the new concept and will copy over all the properties, references and children of the old node to the new node without checking whether they are appropriate for the new concept. This may or may not be what you want. It will also fix the references in the descendants that point to the original node to point to the new one.

If this behavior does not suit you, you can manually create a node with the specified ID and copy over the necessary features.

Setting the ID of a node after creation

We cannot set the ID of a node via Open API but we can cast the org.jetbrains.mps.openapi.model.SNode interface to jetbrains.mps.smodel.SNode class and use its setId() method:

((SNode) newNode).setId(oldNode/.getNodeId());

The node must not be part of any model at the moment of the call to setId().

Setting the ID of a node in a quotation

If we create nodes using a quotation (light or full), we can use a hidden feature to specify the ID of the newly created node. Typing a left square bracket ([) after a quotation will pop up two extra fields where we can specify the model to use when creating the node and the ID of the new node.

(The model is a technical parameter that can be simply set to null. Specifying a model here does not add the node to the model after creation, it simply asks the model to create the node, but all current model implementations create nodes in the same way anyway.)

Fixing the references

The replaceWithNewConcept() method described previously uses this code to fix reference targets in the new node:

Conclusion

We cannot change the concept of a node directly but there is a workaround that works. Just don’t do it during generation. (But you probably won’t need to, anyway.)

We could do it by creating and showing a Swing dialog, but we can also create a concept to store the configuration in the model and invoke the importer right from the editor of that concept by clicking a button.

Here is how it could look like:

Why editor?

Why use the editor instead of an action? By creating a separate configuration concept we can easily persist a configuration so that the user does not need to re-enter it every time she wants to run an import. We can also store multiple alternative configurations. An MPS editor is also easier to design than a Swing dialog.

Implementation

We create a separate language for import configuration, game.importconfig, to keep the core language clean of infrastructure concerns. The configuration is represented by a single rootable concept, ImportConfiguration, with a string property for the source file.

The editor of ImportConfiguration embeds a Swing button to invoke the import logic. The code for the button is shown below:

The implementation is similar in spirit to the action implementation from the previous posts:

• We retrieve the current IDEA project using the same mechanism (DataManager and DataContext) that IDEA actions use internally.
• The actionPerformed method will be invoked outside of any model access lock so we need to start a command manually using the command statement from jetbrains.mps.lang.access language.
• Any exception thrown is shown using a notification, same as in the action.

Find the latest code on GitHub. The game.sandbox.imported model contains a sample import configuration.

You can pass options to git clean to exclude all of the above, but the command becomes cumbersome:

git clean -dx -e .idea -e .mps -e '*.properties' -e '*.log'

At some point I defined an alias for the above command:

alias mpsclean="git clean -dx -e .idea -e .mps -e '*.properties' -e '*.log'"

For safety reasons running the command above will not actually do anything as it’s missing a -f or -n flag that would tell it to either delete the files (-f) or do a dry run, listing all the files it is going to delete (-n). The flag needs to be provided explicitly. So I would use the alias like this:

mpsclean -n # do not delete anything yet
# check the list of files and directories to be deleted
mpsclean -f # actually delete the files

Each menu in MPS shows a particular action group. Action groups are extensible, dynamic lists of actions whose contents can be modified by plugins such as our importer plugin. They are also recursive: you can add an action group to another action group.

In fact, you cannot add a single action directly to an action group but have to wrap it in an action group of its own first.

Finding the right group

The biggest stumbling block here is finding the right group to add the action to. Although the documentation states that the predefined MPS groups are stored in the jetbrains.mps.ide.actions model, it is not really helpful in this instance.

First, the documentation is incomplete. For example, the EditorPopup group that represents the context menu in an editor is defined in jetbrains.mps.ide.editor.actions (note the .editor. in the name).

Second, jetbrains.mps.ide.actions model contains dozens of actions, and the documentation does not specify which group corresponds to which menu.

We can try guessing the name or looking through the completion menu. In our case we look for the context menu related to a model so we could try typing Model and see what comes up.

If that fails, we can look up an action that is already present in the menu, and use Find Usages to find which groups contain it. Ideally the action should be something distinct that would be easy to track, as opposed to a generic action like New or Build Model.

Let’s say we look through the menu and find an action called Analyze Model Dependencies. This is the action’s title but we need to find the actual action node whose name is going to be a Java identifier. So we need to guess again.

In this case, AnalyzeModelDependencies doesn’t bring up any results but searching for just Analyze finds AnalyzeDependencies which turns out to be the action group we are looking for.

The next step is to search for usages of AnalyzeDependencies, globally. To do that we bring up the Find Usages Settings dialog by pressing Ctrl+Alt+Shift+F7, change the Scope to Global and confirm.

This brings us to ModelActions group that is the group we are looking for. It mostly consists of anchors (items with an arrow in front of their names) that represent sections that other action groups can add actions to, to keep related actions together.

Adding the action to the group

After we have figured out the action group, adding the action is fairly straightforward. I have recorded a screencast of the necessary steps:

Next: invoking importer from an editor

In the next post we will implement an alternative way of invoking importers: from an editor. Meanwhile, the latest code is on GitHub.

I have started looking at the MPS Coderules plugin which implements an alternative analysis engine for MPS based on Constraint Handling Rules (CHR), a rule-based declarative language, roughly occupying the same position in whatever programming language taxonomy as does Prolog.

Coderules may be the future of MPS type checking now that the Kotlin-for-MPS type system is being written in it. That’s why I wanted to get a feeling for how easy it would be to understand an existing type system or develop one.

The shock

My first reaction when I saw a code rules program was that of resistance and recoil: the documentation is full of similar-sounding foreign terms like rule, constraint, query, or macro. It appears that the language does not simply describe the rules of typechecking but rather rules that build constraints. (Which seemed to be other kinds of rules. After all isn’t a constraint a kind of rule that constrains something?)

This was all a bit too meta to me so I quit.

The next time I tried looking at Kotlin’s typesystem linked above but was immediately lost and overwhelmed. So I quit a second time.

After several such attempts I started recognizing some patterns and making some distinctions.

I realized that rules are programs but constraints are not. Constraints are simply special objects floating around in the reactor (the working memory) of the engine. The engine is processing the constraints according to the rules: it matches them to a rule, the rule tells which of the matched constraints to keep and which to remove, and which new constraints to add.

To keep with the reactor analogy, constraints are molecules and rules are chemical reactions that transform some molecules into other molecules. Some of the “input” molecules are consumed while other molecules work like catalysts, enabling the reaction while not being consumed in the process.

Entry and exit point

Past the initial shock, my next immediate question was “Where does this program start and how does it end?” The entry point seems to be a query, which the type checker calls when it wants to know the type of something. There are multiple queries (check and typeOf) and I don’t yet understand the difference between them.

You can define queries (or, rather, query handlers) in two ways: using a lower-level QueryTable concept, or, for the type system, using TypecheckingQueries concept. The latter is where we finally get a little bit domain-specific, as the underlying engine is so general that besides computing types it can apparently also compute control flow, data flow and other kinds of analyses).

The exit point is probably when the reactor “stabilizes”, i.e. no new rules can be fired. I have expected that the engine would extract the “answer”, i.e. the type, from the constraints that remain in the reactor somehow but this does not seem to be the case. Instead, the query (the entry point) gets passed a type collector which collects types via side effects:

(I am still a bit confused here because the method to retrieve the type collector from a query is deprecated, yet I see no indication of there being a replacement.)

To be continued…

This is all that I have time to write today but I want to continue my investigation of coderules in the future. I still have lots of open questions, both low and high level, but I am curious what questions you might have about Coderules.

File Chooser

Since MPS builds upon the IDEA Platform which builds upon Swing, we can use the usual Swing components as well as the platform-provided facilities. I prefer to use classes from the platform when available because they often provide a more polished look-and-feel. In this case the platform provides a class called FileChooser for opening a “choose file” dialog, documented here.

Projects

To invoke the FileChooser we will need to pass to it the current project instance. There are in fact several different project classes in MPS, one coming from IDEA (com.intellij.openapi.project.Project) and another one coming from MPS (jetbrains.mps.project.MPSProject). In fact, there are other project classes in MPS but we will not need to concern ourselves with them in this tutorial.

To obtain an instance of the IDEA project we add a context parameter of type Project to our action. I like to call it ideaProject to disambiguate it from a MPS project.

Here is what the FileChooser invocation looks like:

VirtualFile chosenFile = FileChooser.chooseFile(
  FileChooserDescriptorFactory.createSingleFileDescriptor("json"), this.ideaProject, null);

Virtual file system

The result we get from the dialog is a VirtualFile object. Virtual files are abstractions provided by the IDEA platform’s virtual file system (VFS). We are not going to use VFS in our code though, so we will escape the abstraction immediately by calling getPath() on it.

Next, we need to make sure that we are not calling the file chooser while holding a model lock.

Model access: reading, writing, and commanding

All model access in MPS takes place under a read-write lock. The rules are documented in the MPS help. In our case, the importer modifies the model so it will need to be invoked from within a command.

By default, an action’s execute block is wrapped in a command automatically and we don’t have to think of this. On the other hand, UI calls are disallowed when under a read-write lock to prevent UI freezes and deadlocks. This forces us to manage model locking explicitly in our code so we need to disable the automatic command by changing required access from command to none.

If we now need to execute a piece of code in a command, we do it explicitly by using the command statement from jetbrains.mps.lang.access language that was helpfully added to the model for us by the wizard that created the plugin solution.

We need to pass a repository to the command, which we can obtain from the current MPS project, which we get from the context like the IDEA project above.

Show error messages as balloons

Our importer logic can throw exceptions that we should show to the user. If we don’t do anything and simply rethrow the exception, it will cause a flashing exclamation mark to appear in the bottom right corner. If we catch the exception we can use a plain Swing message box to show it, or, even better, a notification. I have written previously about notifications here.

The complete code

Here is what the action looks like after all the modifications:

Next: adding the action to a menu

The action works now but can still only be invoked via the Find Action popup (Ctrl/Cmd+Shift+A). In the next post we will add it to a menu.

A common way of extending MPS is via plugins that provide actions. An action is a piece of code that can be invoked via a menu, such as the main menu or a context menu, or a key combination. In the context of MPS it is important to distinguish these so called plugin actions from editor actions which are invoked by the MPS editor to manipulate the node being edited. Both types of actions can be provided by the language developer but plugin actions can perform any task and have access to the entire IDE, whereas editor actions only have access to the current editor and the node being edited.

To launch the importer our action will need two pieces of input from the user: the source file and the target model. Our plan will be to take the target model from the context and use the built-in file chooser dialog to ask for the source file.

Plugin solution

Whereas we were free to put our Java code (almost) anywhere we like, we are more constrained with plugins and actions. They must reside in a specially configured solution in a specially named model. MPS provides a wizard that can create both the solution and the model for us, found in the context menu of the project (New → Plugin Solution):

The wizard will perform three tasks for us:

1. Create a solution and set its Solution Kind set to Other. You can also manually set the solution kind on the Java tab in the solution properties dialog:

   

2. Create a model named <solution-name>.plugin with a few useful languages already imported for use.

3. Create a root node of concept StandalonePluginDescriptor.

These three arcane bits of configuration are required for MPS to load actions and other plugin elements from the solution. More details about MPS classloading and solution kinds can be found on a helpful Wiki page of the mbeddr project.

Creating an action

We can now create our first action in the model (right click on the model → New → j.m.lang.plugin → Action). Let’s call it ImportLocationsFromJson and set the caption to “Import Locations from JSON…”. The ellipsis at the end is a convention to indicate that the action will show a dialog.

As a quick check that things so far have been configured properly, after we build the action we should be able to use the “Find Action” popup (Ctrl/Cmd+Shift+A) to look it up:

You can even try invoking it from that popup (press Enter): it should do nothing at all (including throwing no errors).

NOTE: Actions are documented in the JetBrains MPS Help and I suggest you read the documentation to gain a better understanding of what we are going to do.

Getting the target model: context parameters

Our action will be placed in a context menu of a model. It can determine the model that it is invoked on via a context parameter with key MODEL. To use this key we need to import the model that contains MPSCommonDataKeys class, jetbrains.mps.ide.actions@java_stub:

If MPS does not give the name to the parameter automatically, name it model. Leave it required.

Adding a required parameter serves two purposes: we will be able to use it in the execute block, and the action will be disabled if invoked from a context where no model is present. If you now try looking up the action again via Cmd/Ctrl+Shift+A, you will find that the action is disabled if the editor is focused and enabled if the focus is on a model in the Logical View (project tree).

Next: code!

Having the model means we have one half of our inputs without writing any code yet. In our next part we will have no such luxury and will get our hands dirty with Java code to let the user pick the input file and perform the action. Meanwhile, the code is on GitHub and I’m eager to hear your reactions and thoughts about the tutorial so far.

Continuing from our last post, let’s have a brief look at the data and the language we are going to work with and set things up. Remember, the code is available on GitHub.

The language

I have created a very small language for this tutorial, called game.core. It consists of a single concept named Location, representing a location in a game:

In text-based adventures each location is connected to other locations, for example via compass directions (north, east, south, west). To keep things simple but still interesting in our case, each location has just one optional next reference pointing to the next location in the adventure.

Each location also has a name (inherited from INamedConcept) and a description. A location is uniquely identified by its name. The description field holds additional information about the location.

The input

We will be importing JSON data such as this file (available in the GitHub repository as input/adventure.json:

{
    "locations": [
        {
            "name": "Street",
            "description": "You stand in front of an old house that looks just like you remembered it in the past. The entrance door is ajar.",
            "next": "House Entrance"
        },
        {
            "name": "House Entrance",
            "description": "You enter the house. Everything here is covered with dust: the floor, the hanger, the shelves. Except for one dust-free door with a label on it that says \"Cellar\".",
            "next": "Cellar"
        },
        {
            "name": "Cellar",
            "description": "The cellar is dark and you can't see anything. Your right hand rests on the wall and senses a light switch."
        }
    ]
}

Note that a location refers to its next location by its (unique) name. Also note that we can refer to locations that appear later in the file (i.e. the importer must support forward references).

Gson

To parse JSON data we will use the Gson library from Google. To do that we will need to add the Gson JAR to our project. The process of doing so is well documented elsewhere and I will not cover it here. In the linked project

Now we can test our parser from the MPS Console:

> {
    JsonElement element; 
    try (Reader reader = Files.newBufferedReader(Path.of("/path/to/sample-importer/input/adventure.json"))) { 
      element = JsonParser.parseReader(reader); 
    } 
    #print element;
  }
  // output:
  // {"locations":[{"name":"Street","description":"You stand in front of an old house ...

Since it can be a bit difficult for newcomers to MPS to figure out how to type things properly, here is a screencast of me typing this code into MPS Console and running it:

Where to put the code

The code of the importer will live in a Java class but where do we put the class? We have several options, ordered by the amount of effort required:

1. Put the class in one of the existing aspects of the game.core language.
2. Create a separate model in the game.core and put the class there. Such a model would be called “utility model” in MPS parlance.
3. Create a separate solution for the importer logic, create a model there and put the class into it.

We don’t need to agonize too much about the decision because we can always move classes between models and modules. Since none of the existing language aspects feel appropriate, we create a new model, name it game.core.importer, and a new class ImporterLogic inside:

Initial code

We are now finally ready to write some code! Since our input data contains forward references, we store all nodes in a hash map, keyed by their names. If we encounter a reference to a node we haven’t seen yet, we create it but only initialize its name. Later when we encounter the full data for the target location, we take the previously created node from the map and fill it in.

Here is the full code:

public class ImportLogic {

  public static void importData(string sourceFile, model targetModel) throws IOException {
    final JsonElement element;

    try (Reader reader = Files.newBufferedReader(Path.of(sourceFile))) {
      element = JsonParser.parseReader(reader);
    }

    JsonArray locationsArray = element.getAsJsonObject().get("locations").getAsJsonArray();

    // Store locations in a map to resolve forward references
    Map<string, node<Location>> locationsByName = new HashMap<string, node<Location>>();

    foreach locationElement in locationsArray {
      JsonObject locationObject = locationElement.getAsJsonObject();

      string name = locationObject.get("name").getAsString();
      string description = locationObject.get("description").getAsString();
      JsonElement nextElement = locationObject.get("next");
      string nextName = nextElement == null ? null : nextElement.getAsString();

      node<Location> location = locationsByName.computeIfAbsent(name, {string name => <Location(name: name)>; });

      node<Location> nextLocation = nextName == null ? null : locationsByName.computeIfAbsent(nextName, {string name => <Location(name: name)>; });

      location.description = description;
      location.next = nextLocation;

      targetModel.add root(location);
    }
  }
}

Running the importer

We don’t have any user-friendly way to run the importer yet (I will cover it in a later post) but we can already run it from the MPS Console. Before we do that we need to create a target model for the imported nodes. Let’s put it into the game.sandbox solution and name it game.sandbox.imported. Next, we can invoke the importer from the console as follows:

The console shows an error because we don’t catch IOException but the code will compile and run. As a result, you should see the three locations created in the game.sandbox.imported model.

Note that since the code in ImportLogic#importData() manipulates the target model, it must run in the context of a command. MPS Console establishes a command for us automatically, in other cases we may need to start a command manually.

In the next post we will look at invoking the importer in a user-friendly way. Until then, I would be happy to get any feedback and questions you have.

That’s why I want to start a series of beginner-level posts to cover the process of writing an importer. I will assume you have experience programming in Java but are just starting with MPS.

Our domain is going to be text-based adventure games, also known as interactive fiction. We will use a very simple language for describing a very simple adventure game, and import game data from a JSON file into MPS. I am not going to cover the development of the language itself in this series and will only focus on the importing process.

I will be using the latest currently released version of MPS, 2021.2.3, and I will publish my code on GitHub as we go, so you can follow along. The linked repository already contains our simple language, a sandbox where I have manually created a few game locations, and a sample JSON input file with the map data. I will talk more about the language in the next post.

In the upcoming posts I am going to cover these topics:

• The basics. Using Google Gson from MPS to read a JSON file and using the MPS SModel API to create MPS nodes from the data.
• Invoking the importer. Various ways in which you can launch the import from within MPS.
• Supporting subsequent reimports. How to update the model when the input data changes.
• Error handling. Letting the user know if there was a problem with the input data.

The list of topics isn’t set in stone and I may come up with additional topics as we go. But if there’s an interesting aspect related to importers that you want me to cover, let me know. You can hit Reply if you are reading this as an email or email me directly at sergej@specificlanguages.com).