Tooling: Brainstorming ideas that can lead to efficient loader-oriented designs

[SMotaal]

Having both ecmascript-modules and @jkrems hackable loader has opened up tremendous scope for experimentation.

Note: This thread does not make claims for or against existing tooling, some of which have stood the test of time, evolved, and are fixtures of the ecosystem. The intent is simply to consider different perspectives being explored in experimental efforts.

As far as things go, the broad range of tooling that applies to loaders basically iterates over productions in each source, irrespective of the specifics of implementation or operations.

Most tools are designed to be used for much more complex applications than merely loading. To that effect, they often avoid the use of new language features that would prevent them from working on older platforms. They can also avoid new features which may have been prematurely associated with inefficiencies in early stages. Some are also built with infrastructures or features that are not ideal or not optimized specifically for loading, like using workers, verbose error checking (ie as a language service)... etc.

I would like to dedicate this thread to brainstorming experimental or just different ideas to implement related patterns for loader-first designs.

Brainstorming: A safe place to discuss ideas and provide constructive feedback

How to contribute

Please avoid emoting that can be confusing (especially if it can construed as passively aggressive)

😄	Indication
👍	To indicate a "Yes" response
👎	To indicate a "No" response
🎉	To indicate a "Aha" moment

---

Read the Digest

The following is a set of ideas or conclusions curated from the discussions:

Syntax Detection (CJS vs ESM)

• Safely using RegExp — @SMotaal

  • requires guarding against string hijacking — @devsnek
  • recommend using acorn instead — @devsnek

• Fallback for ESM without import and export — @targos

  • shouldn't use import(…) to resolve ambiguity — @bmeck
  • can use import.meta — @bmeck

• Dual parsing a module was deemed inefficient — @MylesBorins

  • consider @jdalton's top-level parse — @bmeck
  • consider bailing-early — @SMotaal
  • consider using single comparative scans — @SMotaal

Syntax Identification (CJS vs ESM)

• Mime type meta data via something like webpackage — @jkrems

  • may rely on package.json — @GeoffreyBooth
  • may borrow from the idea of service-worker scopes — @SMotaal

• Magic bytes — @jkrems

Wrapping CJS in an ESM module system

• Sometimes CJS is not "use strict" — @benjamingr
  • requires evaluating wrapper in non-strict — @SMotaal
  • can use multiple SourceTexts — @bmeck

[SMotaal]

ECMAScript modules syntax can arguably be detected using a RegExp which bails on first match.

Does anyone have ideas for cjs vs esm syntax detection?

[devsnek]

@SMotaal you can't use regexp to parse js grammar (you can always make a pattern of string literals or whatever to confuse the regexp) and the differences between valid cjs and valid esm are ambiguous and can't be reliably detected by just looking at the code.

[SMotaal]

you can always make a pattern of string literals or whatever to confuse the regexp

So, can we constructively say that so long as you guard against string hijacking (maybe there is a better term for this), only then can you safely use RegExp?

[devsnek]

@SMotaal I would just use acorn

[SMotaal]

differences between valid cjs and valid esm are ambiguous and can't be reliably detected by just looking at the code

Can you elaborate on this please?

[SMotaal]

@devsnek humor me in this effort, consider this both an idea-gathering as well as a team-building exercise. Acron is obviously a great solution, but I am trying to create opportunities for people to talk about the aspects that make this and others such great tools. The notion here is that people might just have some evolving ideas that they might want to bounce around. How we connect the dots, like you pointing out the hijacking limitation can potentially inspire untapped solutions to existing problems.

Sounds fair?

[targos]

@SMotaal You could say that a file with import or export syntax is probably an ES Module (the syntax is invalid in Script mode). However, the problem is that files without import and export could be either Script or Module, and depending on how they are written, could have different behaviour in Script vs Module mode.

For example:

test = 42;

In Script mode, this creates the property test on the global object.
In Module mode, this throws a ReferenceError.

[benjamingr]

@targos does the issue get any better if we say that such a loader always imports CJS in strict mode regardless of an explicit "use strict"?

[devsnek]

are we trying to come up with use cases for loaders or something else?

if you're using a resolve loader hook you'll always be able to read the contents of whatever you're resolving, at which point you can regex or acorn or whatever it as you see fit.

[targos]

I'm having trouble to see the relation between 'cjs vs esm syntax detection" and the OP. Maybe I don't really understand what this thread is about, sorry.