If we run the importer twice in its current state, we get the imported nodes duplicated. Simply clearing the model before an import and recreating it during an import would avoid this problem, but would break any existing references to the previously imported nodes, and this is usually not ideal.

Thus, we are left with trying to find existing nodes in the model and updating them. To be able to do that, we need to know whether a node matches an existing one. To do that, each such node has to carry a domain-specific unique identifier.

Note that not every node has to have an identifier, only nodes that may be referenced from outside the import. The identifiers don’t have to be unique globally, usually uniqueness within a single concept (or item type) is enough.

Previously we have modified our importer to add all nodes to a temporary model first, and add them to the original model in a separate step. We did this originally to avoid leaving the model in incosistent state on failure, but this two-phase approach will also help us with our today’s task.

We will modify the second step to “synchronize” or “merge” the temporary model into the target model, rather than blindly add nodes.

The merge algorithm is an area that may become quite complicated depending on what kinds of changes you want to handle between reimports. In our simple case it is enough to look through the model for an existing node and update it if it exists, or create a new one:

(The code can be found in the usual place on GitHub.)

This code can also be optimized by building a map of existing nodes rather than searching through the model each time. Otherwise, the simplistic approach works quite well. However, depending on your needs, you may need to modify your merge algorithm to provide answers to these questions:

• What do we do with nodes that are present in the original model but not in the import? If these nodes are not supposed to be referenced from outside the model, it is safe to delete them. Another option could be to mark them as hidden or removed with a special boolean property. Or maybe even “commenting them out” using MPS universal comments.

• Is the order of certain child nodes within their parent important? If so, the child list may have to be rearranged according to the incoming model.

• What about forward references to child nodes, i.e. referring to a child node that does not exist yet. In this case you might not be able to add it to the model when you encounter it (like we do in findOrCreateLocation in our example) because its parent may not exist yet. You could perhaps hold them in some kind of a “buffer” first, and at the end, check that the “buffer” nodes have all been added to the model (because they were visited from other nodes).

These considerations are shared among many importers, and Daniel Binkele-Raible from itemis has been working on a generic framework for merging models. It is currently undergoing review before it can be merged into MPS-extensions but you can see the current state in this pull request.

1. The input file can be unparseable as JSON. For example, the user has chosen a .zip file by mistake instead of .json.
2. A valid JSON file can have unexpected structure. For example, the user has chosen a .json file with a different, unrelated data set.
3. The data set is correct but contains incorrect values. For example, a reference mentions an object that does not exist, a field value is out of expected range, or some references form an unexpected cycle.
4. The file could be fully valid but unexpectedly large.

We often want to avoid leaving the model in an inconsistent state after a failed import. While we can sometimes assume that users are using version control and know how to roll back unintended changes, it is nicer if they don’t have to. Our current importer logic adds the objects straight into the target model as we go through the parsed structure, and as a result it is prone to this issue.

We can fix this by importing objects into a temporary model first, and only moving them to the target model once the full import is complete. While in theory the move could partially fail as well, in practice this is unlikely as it all happens in memory with data that has passed validation.

To create a temporary model we can make use of jetbrains.mps.smodel.tempmodel.TemporaryModels singleton:

TemporaryModels temporaryModels = TemporaryModels.getInstance();
model tempModel = temporaryModels.createReadOnly(TempModuleOptions.forDefaultModule());
try {
  importDataIntoModel(sourceFile, tempModel);

  foreach root in tempModel.roots(<all>) {
    targetModel.add root(root);
  }
} finally {
  temporaryModels.dispose(tempModel);
}

I have moved the original code into importDataIntoModel method that we now call to import the data into the temporary model. After the import is successful, we add all roots to the target model. Adding a root to another model removes it from its original model.

When creating a temporary model we have to choose whether it will be read-only or editable. This pertains to the ability of the user to edit it via an editor, it is possible to modify the model via code in any case. We do not show the model to the user and in any case we don’t intend it to be modifiable by the user, so we choose the read-only option.

In the next post we will make it possible to update already existing data via import, rather than adding new objects on each import. In the meantime, the code is on GitHub.

We could do it by creating and showing a Swing dialog, but we can also create a concept to store the configuration in the model and invoke the importer right from the editor of that concept by clicking a button.

Here is how it could look like:

Why editor?

Why use the editor instead of an action? By creating a separate configuration concept we can easily persist a configuration so that the user does not need to re-enter it every time she wants to run an import. We can also store multiple alternative configurations. An MPS editor is also easier to design than a Swing dialog.

Implementation

We create a separate language for import configuration, game.importconfig, to keep the core language clean of infrastructure concerns. The configuration is represented by a single rootable concept, ImportConfiguration, with a string property for the source file.

The editor of ImportConfiguration embeds a Swing button to invoke the import logic. The code for the button is shown below:

The implementation is similar in spirit to the action implementation from the previous posts:

• We retrieve the current IDEA project using the same mechanism (DataManager and DataContext) that IDEA actions use internally.
• The actionPerformed method will be invoked outside of any model access lock so we need to start a command manually using the command statement from jetbrains.mps.lang.access language.
• Any exception thrown is shown using a notification, same as in the action.

Find the latest code on GitHub. The game.sandbox.imported model contains a sample import configuration.

Each menu in MPS shows a particular action group. Action groups are extensible, dynamic lists of actions whose contents can be modified by plugins such as our importer plugin. They are also recursive: you can add an action group to another action group.

In fact, you cannot add a single action directly to an action group but have to wrap it in an action group of its own first.

Finding the right group

The biggest stumbling block here is finding the right group to add the action to. Although the documentation states that the predefined MPS groups are stored in the jetbrains.mps.ide.actions model, it is not really helpful in this instance.

First, the documentation is incomplete. For example, the EditorPopup group that represents the context menu in an editor is defined in jetbrains.mps.ide.editor.actions (note the .editor. in the name).

Second, jetbrains.mps.ide.actions model contains dozens of actions, and the documentation does not specify which group corresponds to which menu.

We can try guessing the name or looking through the completion menu. In our case we look for the context menu related to a model so we could try typing Model and see what comes up.

If that fails, we can look up an action that is already present in the menu, and use Find Usages to find which groups contain it. Ideally the action should be something distinct that would be easy to track, as opposed to a generic action like New or Build Model.

Let’s say we look through the menu and find an action called Analyze Model Dependencies. This is the action’s title but we need to find the actual action node whose name is going to be a Java identifier. So we need to guess again.

In this case, AnalyzeModelDependencies doesn’t bring up any results but searching for just Analyze finds AnalyzeDependencies which turns out to be the action group we are looking for.

The next step is to search for usages of AnalyzeDependencies, globally. To do that we bring up the Find Usages Settings dialog by pressing Ctrl+Alt+Shift+F7, change the Scope to Global and confirm.

This brings us to ModelActions group that is the group we are looking for. It mostly consists of anchors (items with an arrow in front of their names) that represent sections that other action groups can add actions to, to keep related actions together.

Adding the action to the group

After we have figured out the action group, adding the action is fairly straightforward. I have recorded a screencast of the necessary steps:

Next: invoking importer from an editor

In the next post we will implement an alternative way of invoking importers: from an editor. Meanwhile, the latest code is on GitHub.

File Chooser

Since MPS builds upon the IDEA Platform which builds upon Swing, we can use the usual Swing components as well as the platform-provided facilities. I prefer to use classes from the platform when available because they often provide a more polished look-and-feel. In this case the platform provides a class called FileChooser for opening a “choose file” dialog, documented here.

Projects

To invoke the FileChooser we will need to pass to it the current project instance. There are in fact several different project classes in MPS, one coming from IDEA (com.intellij.openapi.project.Project) and another one coming from MPS (jetbrains.mps.project.MPSProject). In fact, there are other project classes in MPS but we will not need to concern ourselves with them in this tutorial.

To obtain an instance of the IDEA project we add a context parameter of type Project to our action. I like to call it ideaProject to disambiguate it from a MPS project.

Here is what the FileChooser invocation looks like:

VirtualFile chosenFile = FileChooser.chooseFile(
  FileChooserDescriptorFactory.createSingleFileDescriptor("json"), this.ideaProject, null);

Virtual file system

The result we get from the dialog is a VirtualFile object. Virtual files are abstractions provided by the IDEA platform’s virtual file system (VFS). We are not going to use VFS in our code though, so we will escape the abstraction immediately by calling getPath() on it.

Next, we need to make sure that we are not calling the file chooser while holding a model lock.

Model access: reading, writing, and commanding

All model access in MPS takes place under a read-write lock. The rules are documented in the MPS help. In our case, the importer modifies the model so it will need to be invoked from within a command.

By default, an action’s execute block is wrapped in a command automatically and we don’t have to think of this. On the other hand, UI calls are disallowed when under a read-write lock to prevent UI freezes and deadlocks. This forces us to manage model locking explicitly in our code so we need to disable the automatic command by changing required access from command to none.

If we now need to execute a piece of code in a command, we do it explicitly by using the command statement from jetbrains.mps.lang.access language that was helpfully added to the model for us by the wizard that created the plugin solution.

We need to pass a repository to the command, which we can obtain from the current MPS project, which we get from the context like the IDEA project above.

Show error messages as balloons

Our importer logic can throw exceptions that we should show to the user. If we don’t do anything and simply rethrow the exception, it will cause a flashing exclamation mark to appear in the bottom right corner. If we catch the exception we can use a plain Swing message box to show it, or, even better, a notification. I have written previously about notifications here.

The complete code

Here is what the action looks like after all the modifications:

Next: adding the action to a menu

The action works now but can still only be invoked via the Find Action popup (Ctrl/Cmd+Shift+A). In the next post we will add it to a menu.

A common way of extending MPS is via plugins that provide actions. An action is a piece of code that can be invoked via a menu, such as the main menu or a context menu, or a key combination. In the context of MPS it is important to distinguish these so called plugin actions from editor actions which are invoked by the MPS editor to manipulate the node being edited. Both types of actions can be provided by the language developer but plugin actions can perform any task and have access to the entire IDE, whereas editor actions only have access to the current editor and the node being edited.

To launch the importer our action will need two pieces of input from the user: the source file and the target model. Our plan will be to take the target model from the context and use the built-in file chooser dialog to ask for the source file.

Plugin solution

Whereas we were free to put our Java code (almost) anywhere we like, we are more constrained with plugins and actions. They must reside in a specially configured solution in a specially named model. MPS provides a wizard that can create both the solution and the model for us, found in the context menu of the project (New → Plugin Solution):

The wizard will perform three tasks for us:

1. Create a solution and set its Solution Kind set to Other. You can also manually set the solution kind on the Java tab in the solution properties dialog:

   

2. Create a model named <solution-name>.plugin with a few useful languages already imported for use.

3. Create a root node of concept StandalonePluginDescriptor.

These three arcane bits of configuration are required for MPS to load actions and other plugin elements from the solution. More details about MPS classloading and solution kinds can be found on a helpful Wiki page of the mbeddr project.

Creating an action

We can now create our first action in the model (right click on the model → New → j.m.lang.plugin → Action). Let’s call it ImportLocationsFromJson and set the caption to “Import Locations from JSON…”. The ellipsis at the end is a convention to indicate that the action will show a dialog.

As a quick check that things so far have been configured properly, after we build the action we should be able to use the “Find Action” popup (Ctrl/Cmd+Shift+A) to look it up:

You can even try invoking it from that popup (press Enter): it should do nothing at all (including throwing no errors).

NOTE: Actions are documented in the JetBrains MPS Help and I suggest you read the documentation to gain a better understanding of what we are going to do.

Getting the target model: context parameters

Our action will be placed in a context menu of a model. It can determine the model that it is invoked on via a context parameter with key MODEL. To use this key we need to import the model that contains MPSCommonDataKeys class, jetbrains.mps.ide.actions@java_stub:

If MPS does not give the name to the parameter automatically, name it model. Leave it required.

Adding a required parameter serves two purposes: we will be able to use it in the execute block, and the action will be disabled if invoked from a context where no model is present. If you now try looking up the action again via Cmd/Ctrl+Shift+A, you will find that the action is disabled if the editor is focused and enabled if the focus is on a model in the Logical View (project tree).

Next: code!

Having the model means we have one half of our inputs without writing any code yet. In our next part we will have no such luxury and will get our hands dirty with Java code to let the user pick the input file and perform the action. Meanwhile, the code is on GitHub and I’m eager to hear your reactions and thoughts about the tutorial so far.

Continuing from our last post, let’s have a brief look at the data and the language we are going to work with and set things up. Remember, the code is available on GitHub.

The language

I have created a very small language for this tutorial, called game.core. It consists of a single concept named Location, representing a location in a game:

In text-based adventures each location is connected to other locations, for example via compass directions (north, east, south, west). To keep things simple but still interesting in our case, each location has just one optional next reference pointing to the next location in the adventure.

Each location also has a name (inherited from INamedConcept) and a description. A location is uniquely identified by its name. The description field holds additional information about the location.

The input

We will be importing JSON data such as this file (available in the GitHub repository as input/adventure.json:

{
    "locations": [
        {
            "name": "Street",
            "description": "You stand in front of an old house that looks just like you remembered it in the past. The entrance door is ajar.",
            "next": "House Entrance"
        },
        {
            "name": "House Entrance",
            "description": "You enter the house. Everything here is covered with dust: the floor, the hanger, the shelves. Except for one dust-free door with a label on it that says \"Cellar\".",
            "next": "Cellar"
        },
        {
            "name": "Cellar",
            "description": "The cellar is dark and you can't see anything. Your right hand rests on the wall and senses a light switch."
        }
    ]
}

Note that a location refers to its next location by its (unique) name. Also note that we can refer to locations that appear later in the file (i.e. the importer must support forward references).

Gson

To parse JSON data we will use the Gson library from Google. To do that we will need to add the Gson JAR to our project. The process of doing so is well documented elsewhere and I will not cover it here. In the linked project

Now we can test our parser from the MPS Console:

> {
    JsonElement element; 
    try (Reader reader = Files.newBufferedReader(Path.of("/path/to/sample-importer/input/adventure.json"))) { 
      element = JsonParser.parseReader(reader); 
    } 
    #print element;
  }
  // output:
  // {"locations":[{"name":"Street","description":"You stand in front of an old house ...

Since it can be a bit difficult for newcomers to MPS to figure out how to type things properly, here is a screencast of me typing this code into MPS Console and running it:

Where to put the code

The code of the importer will live in a Java class but where do we put the class? We have several options, ordered by the amount of effort required:

1. Put the class in one of the existing aspects of the game.core language.
2. Create a separate model in the game.core and put the class there. Such a model would be called “utility model” in MPS parlance.
3. Create a separate solution for the importer logic, create a model there and put the class into it.

We don’t need to agonize too much about the decision because we can always move classes between models and modules. Since none of the existing language aspects feel appropriate, we create a new model, name it game.core.importer, and a new class ImporterLogic inside:

Initial code

We are now finally ready to write some code! Since our input data contains forward references, we store all nodes in a hash map, keyed by their names. If we encounter a reference to a node we haven’t seen yet, we create it but only initialize its name. Later when we encounter the full data for the target location, we take the previously created node from the map and fill it in.

Here is the full code:

public class ImportLogic {

  public static void importData(string sourceFile, model targetModel) throws IOException {
    final JsonElement element;

    try (Reader reader = Files.newBufferedReader(Path.of(sourceFile))) {
      element = JsonParser.parseReader(reader);
    }

    JsonArray locationsArray = element.getAsJsonObject().get("locations").getAsJsonArray();

    // Store locations in a map to resolve forward references
    Map<string, node<Location>> locationsByName = new HashMap<string, node<Location>>();

    foreach locationElement in locationsArray {
      JsonObject locationObject = locationElement.getAsJsonObject();

      string name = locationObject.get("name").getAsString();
      string description = locationObject.get("description").getAsString();
      JsonElement nextElement = locationObject.get("next");
      string nextName = nextElement == null ? null : nextElement.getAsString();

      node<Location> location = locationsByName.computeIfAbsent(name, {string name => <Location(name: name)>; });

      node<Location> nextLocation = nextName == null ? null : locationsByName.computeIfAbsent(nextName, {string name => <Location(name: name)>; });

      location.description = description;
      location.next = nextLocation;

      targetModel.add root(location);
    }
  }
}

Running the importer

We don’t have any user-friendly way to run the importer yet (I will cover it in a later post) but we can already run it from the MPS Console. Before we do that we need to create a target model for the imported nodes. Let’s put it into the game.sandbox solution and name it game.sandbox.imported. Next, we can invoke the importer from the console as follows:

The console shows an error because we don’t catch IOException but the code will compile and run. As a result, you should see the three locations created in the game.sandbox.imported model.

Note that since the code in ImportLogic#importData() manipulates the target model, it must run in the context of a command. MPS Console establishes a command for us automatically, in other cases we may need to start a command manually.

In the next post we will look at invoking the importer in a user-friendly way. Until then, I would be happy to get any feedback and questions you have.

That’s why I want to start a series of beginner-level posts to cover the process of writing an importer. I will assume you have experience programming in Java but are just starting with MPS.

Our domain is going to be text-based adventure games, also known as interactive fiction. We will use a very simple language for describing a very simple adventure game, and import game data from a JSON file into MPS. I am not going to cover the development of the language itself in this series and will only focus on the importing process.

I will be using the latest currently released version of MPS, 2021.2.3, and I will publish my code on GitHub as we go, so you can follow along. The linked repository already contains our simple language, a sandbox where I have manually created a few game locations, and a sample JSON input file with the map data. I will talk more about the language in the next post.

In the upcoming posts I am going to cover these topics:

• The basics. Using Google Gson from MPS to read a JSON file and using the MPS SModel API to create MPS nodes from the data.
• Invoking the importer. Various ways in which you can launch the import from within MPS.
• Supporting subsequent reimports. How to update the model when the input data changes.
• Error handling. Letting the user know if there was a problem with the input data.

The list of topics isn’t set in stone and I may come up with additional topics as we go. But if there’s an interesting aspect related to importers that you want me to cover, let me know. You can hit Reply if you are reading this as an email or email me directly at sergej@specificlanguages.com).