Making Apache POI work with MPS 2020.3: solving class loader hell

One of the developers at one of my clients had a problem today. They couldn’t get Apache POI to work with MPS 2020.3 due to class loading issues. Attempting to run code that used POI would cause this exception (reformatted and abridged for clarity):

java.lang.ExceptionInInitializerError
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderImpl.build(SchemaTypeLoaderImpl.java:161)
	at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.<init>(SchemaTypeSystemImpl.java:198)
	at org.apache.xmlbeans.metadata.system.sXMLSCHEMA.TypeSystemHolder.<init>(TypeSystemHolder.java:9)
	at org.apache.xmlbeans.metadata.system.sXMLSCHEMA.TypeSystemHolder.<clinit>(TypeSystemHolder.java:6)
    ...
Caused by: org.apache.xmlbeans.XmlRuntimeException: java.lang.ClassCastException:
  class org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl
  cannot be cast to
  class org.apache.xmlbeans.SchemaTypeLoader
  (org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl is
    in unnamed module of loader com.intellij.util.lang.UrlClassLoader @eec5a4a;
  org.apache.xmlbeans.SchemaTypeLoader is
    in unnamed module of loader jetbrains.mps.classloading.ModuleClassLoader @35e248b4)
	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderImpl.build(SchemaTypeLoaderImpl.java:164)
	at org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl.<init>(SchemaTypeSystemImpl.java:198)
	at org.apache.xmlbeans.metadata.system.sXMLTOOLS.TypeSystemHolder.<init>(TypeSystemHolder.java:9)
	at org.apache.xmlbeans.metadata.system.sXMLTOOLS.TypeSystemHolder.<clinit>(TypeSystemHolder.java:6)
	... 96 more
Caused by: java.lang.ClassCastException: class org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl cannot be cast to class org.apache.xmlbeans.SchemaTypeLoader (org.apache.xmlbeans.impl.schema.SchemaTypeSystemImpl is in unnamed module of loader com.intellij.util.lang.UrlClassLoader @eec5a4a; org.apache.xmlbeans.SchemaTypeLoader is in unnamed module of loader jetbrains.mps.classloading.ModuleClassLoader @35e248b4)
	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderImpl.build(SchemaTypeLoaderImpl.java:162)
	... 99 more

This problem is caused by having two different versions of Apache XMLBeans library on the classpath at the same time. MPS 2020.3 comes with XMLBeans version 2 in its lib directory whereas the latest version of POI depends on XMLBeans 5.

When XMLBeans initializes itself, it loads some of its own classes via reflection. Unfortunately, the code in version 5 also looks for classes that are not present in this version but were present in XMLBeans 2.

Here is what the class loader hierarchy looks like:

The class loading precedence (“delegation”) rules are such that a class from XMLBeans version 5 will shadow the same class that was present in version 2, and it will be loaded from the module class loader. Its base class will be subject to the same rules and could be loaded from version 5 as well, if present. However, a class that was only present in version 2 would be delegated to the IDEA class loader. The base class of that class would have to be loaded from the same class loader. And so we get two classes that have the same base class but loaded twice from different class loaders. This makes the two base classes incompatible and leads to the exception above.

To make things work, we would need to completely remove the old classes from the class path. To do that, we could upgrade the XMLBeans version that comes bundled with MPS. But this may cause MPS to break due to some incompatibility between the versions. A better solution is to create a separate class loader that would only load the JARs we want and instead of delegating to the IDEA class loader it would delegate directly to the system class loader of the JVM. The hierarchy will then look like this:

Here is how we can create such a custom class loader, making use of IDEA and MPS APIs:

set<String> libraries = new hashset<String>; 
 
libraries.addAll(getClassPathOfModule(module-reference/org.apache.poi/)); 
libraries.addAll(getClassPathOfModule(module-reference/com.spclngs.samples.poi/));

UrlClassLoader classLoader = UrlClassLoader
  .build()
  .parent(ClassLoader.getSystemClassLoader())
  .urls(libraries.select({~it => fileToUrl(it); }).toList)
  .get();

UrlClassLoader is IDEA’s improved version of java.net.URLClassLoader from the JDK. We could have used the latter, too.

The getClassPathOfModule helper function returns the path to the compiled code of the module plus any Java libraries configured for the module on the corresponding tab of its properties. Its code is shown below:

private static Set<String> getClassPathOfModule(SModuleReference moduleRef) { 
  return moduleRef.resolve(MPSModuleRepository.getInstance())
    .getFacet(JavaModuleFacet.class)
    .getClassPath(); 
}

Finally, fileToUrl is a simple helper method to convert a java.io.File to a java.net.URL.

After we create the class loader we can use it to load the class and invoke the method that calls the POI API via reflection:

Class<?> theClass = Class.forName(WriteDocxFile.class.getName(), true, classLoader); 
Method theMethod = theClass.getMethod("writeDocxFile", File.class);
theMethod.invoke(null, outputFile);

And this is the gist of it.

There is a catch, however. The custom class loader that uses POI does not see any MPS classes. This means that it cannot use any SModel APIs or navigate the MPS model. Any data we want to pass to the POI-using code will have to use classes that are visible from both the module class loader and the custom class loader. In our case this means we are limited to basic Java classes such as java.io.File, java.lang.String or collections. Any complex data structure or data from nodes will probably have to be passed as a hash map of hash maps.

Fortunately, it seems that MPS 2021.3 has upgraded its XMLBeans library to version 4 which does not have the extra classes and it should be possible to use Apache POI 5 with MPS 2021.3 without the workaround described above. But the approach may still come handy in case of another version conflict with built-in libraries.

I have published a sample project containing all the code to GitHub.