Proposal: Load foreign code with ES Module Shims and import maps

[thomashoneyman]

Try PureScript is going to have to be reworked when the PureScript compiler switches from CommonJS to ES modules for foreign code. This issue details a proposal for how to handle this rewrite.

Current Implementation

We currently parse through the generated JavaScript on Try PureScript to discover all dependencies present via require() statements. For every dependency we find we do one of:

1. Load the JS from compile.purescript.org (for example, Data.EuclidianRing/foreign.js)
2. Look up its URL from the Shim.purs file which lists CDN locations for some common FFI libraries like react, and then load the contents

Once we know what dependencies to load, we take their contents and insert them into the code iframe. Finally, we load their contents via the evalSources function in frame.js.

Issues

This code is no longer going to work. PureScript 0.15 will only work with ES modules, so the code in Shim.purs, Loader.purs, and frame.js will all need to be updated to work with ES modules instead of CommonJS modules -- in other words, there is no more require or module.exports; instead, there's import and export.

I also don't think we can instantiate the code the same way we do now, which I believe happens here:

trypurescript/client/public/js/frame.js

Line 25 in 53a82a6

new Function("module", "exports", "require", sources[name])(module, module.exports, require);

Finally, if we swap all the way over to ES modules, we lose all Internet Explorer support, which we have historically not wanted to lose.

Proposal: SystemJS

We're clearly going to have to rewrite how we handle foreign code in Try PureScript. After a brief discussion with @natefaubion on the PureScript chat I would like to propose that we use two technologies:

1. SystemJS, which covers IE11
2. Import maps, which are supported in SystemJS, including for IE11

The SystemJS documentation on import maps does a good job of explaining how we can use this tool to replace our existing shim + loader + frame.js setup. If I'm reading correctly, we'd strip out almost everything and move the contents of Shim.purs into an import map. We'd rely on SystemJS for everything else.

[JordanMartinez]

I've spent time trying to understand how client/public/js/frame.js (I'll refer to this as the frame.js) and the scripts in client/public/js/index.html work (I'll refer to this as clientIndex.js). Here's my current understanding:

• clientIndex.js creates the iframe element and tries to send to it the JS outputted by the compiler. It tries 100 times over the course of 10 seconds or stops as soon as frame.js notifies it that the code has been received.
• frame.js loads its DOM, and then receives the JS from clientIndex.js . Once received, it notifies clientIndex.js that code has been received. Then frame.js executes the main function from the outputted JS.
• Whenever the user clicks on an a element in the iframe, frame.js will notify clientIndex.js of the click. If the click is to GitHub repo or gist, then clientIndex.js will allow the redirect. Everything else is blocked for security. See Can't click links in the iframe #137 and Allow gist navigation #140.
• Each time the code is compiled/recompiled, a new iframe is produced.

So, I don't think we can drop frame.js completely due to the GitHub link navigation handling.

As for SystemJS, I've looked at their react-hello-world example to understand how to use it.

Since this project uses the compile end point to also show what the plain JS looks like, we shouldn't change what that outputs. But due to how SystemJS works, I think we need to expose an additional API endpoint (that I'll refer to as compileSystemJs) that is the same as compile but with a few more things. The workflow we need appears to be something like:

• on server setup
  • compile package set's PureScript into JavaScript (src dir -> output dir)
  • use Babel to convert the ES modules into SystemJS modules (output dir -> dist dir)
• on compileSystemJs endpoint:
  • compile user's inputted code to an ES module
  • use Babel to convert that into SystemJS
  • use esbulid to bundle userland SystemJS code and package-set SystemJS code
  • send HTTP response of that output
• on browser side:
  • clientIndex.js forwards the data received by the compileSystemJs endpoint to the iframe
  • iframe sets up and manages whatever is needed for SystemJS
  • SystemJS loads the shims and executes the code

Per their docs, a module can be loaded either by a file or by a module name:

<!-- by file name -->
<script type="systemjs-module" src="/path/to/file.js"></script>
<!-- by module name -->
<script type="systemjs-module" src="import:name-of-module"></script>

We can't load the content by file since the server doesn't and shouldn't store any. But since the iframe is recreated on every compile, we can reuse the same module name. So, I think we'd have to do something like this:

<!-- take the code received by clientIndex.js and
     register it in SystemJS as a module -->
<script>
  // not sure how to do this correctly yet...
</script>
<!-- then execute it -->
<script type="systemjs-module" src="import:main"></script>

The other thing to keep in mind is that the current code will call main in the frame.js part. We would need to add a wrapper around the compiled output that calls main before passing it to Babel, esbuild, and SystemJS.

[JordanMartinez]

First major issue I've encountered is that the plugin for transforming ES module syntax into SystemJS' register format doesn't work on the compiler's output.

[natefaubion]

Per the docs of SystemJS, we probably want https://github.com/guybedford/es-module-shims instead.

[JordanMartinez]

I got es-module-shims to work on PureScript code. See https://github.com/JordanMartinez/use-ems Both decimal.js and big-integer work. I can't verify uuid or the react libraries because the corresponding PS code hasn't been updated to ES modules.

I don't currently have time to integrate this into Try PureScript and submit a PR that implements the needed changes. But I did want to give an update.

One other thing. We'll need to use a CDN that provides NPM deps in the ES module format. We're currently using unpkg, but most modules there seem to be UMD, which don't work with es-module-shims. Per es-module-shims docs and example, we could switch to https://jspm.org/, which can be easily used to generate an import map for the shims. See the repo's dist/index.html for the map and other HTML it generated painlessly.

CC @mikesol

[mikesol]

Thanks for the update!
I'll see if I have time next week to make a PR with these changes.