Add an option to support big-endian machines

[miladfarca]

Hello,

It seems like Big Endian machines are not able to use emscripten, as assertions are added in this commit:
f178541

Runtime error: expected the system to be little-endian!

I understand that there are problems when it comes to "Typed Arrays" in Javascript since they are endianness dependent,
however, WebAssembly is little endian enforced. Runtime compilers such as Google V8 are compatible with running wasm (LE) binary code on BE machines.

Is the above assertion for endianness only done when generating regular js code (such as asm.js) or it
also checks when compiling to WebAssembly? In the case of wasm, would it be possible to continue with LE compilation even on BE machines? (I'm assuming assertions are only done during compilation to JS or to WASM and emscripten isn't concerned with how the generated output gets executed?).

[miladfarca]

Hi @kripken, would you have any suggestions on this issue?

[kripken]

Yes, it is a little odd that typed arrays use the system endianness, while wasm Memories are always little endian.

This is an issue for all emscripten output, because it is a combination of wasm (for compiled code) and JS (to access Web APIs). The JS will use a typed array view to read from memory in a fast way (for example compiled C can call an OpenGL API, and the JS side will read data to send to WebGL). On a big endian machine that may not be correct, hence we have that assertion.

In principle we could add a mode in which we rewrite all memory access from JS to use DataView. We haven't had a request for this yet. Do you have a use case that needs it?

[miladfarca]

Thanks for the info, yes an issue has come up when using eslint-plugin-jest , yarn v2 uses emscripten and this error is generated down the road:
nodejs/citgm#824

So to clarify, it's not possible to always continue with LE ordering when compiling to wasm and only assert when compiling to JS?

[kripken]

@miladfarca Correct. Any time there is any JS - and there is by default, as wasm can't access Web APIs itself - then this is an issue.

The only solutions are either

1. Use standalone mode (no JS, or you write the JS yourself - not recommended if you use any non-trivial amount of JS APIs from the wasm), or
2. Add support for big endian code in emscripten, using DataView.

Option 2 wouldn't be a large amount of work. Is this important for your project?

[miladfarca]

@kripken Thanks for clarifying. It's not a specific project we are trying to support, rather we are trying to support BE architectures in general. Nodejs and V8 support big endian platforms and we are seeing endianness problems with modules similar to eslint-plugin-jest.

In terms of adding support using DataView, is this something we could contribute or emscripten would be able to add the changes and we can test it on BE machines?

[bvibber]

I think some web APIs like WebGL in particular would be more difficult as they'd have to convert data back and forth, knowing which buffers contain what kind of data. However this may be moot, if BE arches are mostly a mainframe/server thing that run node, not browsers.

[miladfarca]

@Brion correct, at this point we are only concerned with Node.js and server related processes and as you mentioned WebGL should not be an issue.

[kripken]

In terms of adding support using DataView, is this something we could contribute or emscripten would be able to add the changes and we can test it on BE machines?

A contribution would be great! It shouldn't be too hard I think. The key thing would be to write a transformation pass on the acorn AST, see for example the growableHeap pass. That pass changes each HEAP8 etc. usage into a function call. For this, I think we'd want to do

HEAP16[..]  =>  dataView.getInt16(.., endianness)

and similar for sets. (The safeHeap pass may also be interesting as it does something very close, but it hasn't been converted to acorn yet.) dataView would need to be created and updated whenever we create/update the HEAP views. And during startup we'd need to detect the endianness. I think that's all of it (well, plus adding an option for this, and a test).

@Brion WebGL would indeed be an issue, yeah. That would come up specifically when doing HEAP*.subarray operations, I think. It's possible we could fix those up just like individual reads. That seems doable, assuming WebGL never receives an Int8Array that it then reads from as a Float32Array, or any other such type mismatch, which I'm not sure of offhand... I think there are packed data structures with multiple types?

[kripken]

Btw, there is a PR up to write SAFE_HEAP using acorn. That replaces HEAP8 etc. usage with a call to a SAFE_HEAP function. An endianness fixup could do something similar (then implement the fixup functions that swap endianness) so that code may be helpful to look at for anyone interested to implement this, #12450

[miladfarca]

@kripken I've created this PR #12780 as a start so we can discuss possible ways to support BE machines. In this case i'm redirecting all the HEAP get/sets with a proxy and using Data Views to do the actual reading and wiring.

I made a simple hello world in C and compiled it in a LE machine with emcc, moved both a.out.wasm and a.out.js to a BE machine and the output is now showing properly. (It was just printing empty lines before this PR, when LE check was commented out).

Note that we still cannot compile with emcc on a BE machine as we get this error:

 Exception message: "SyntaxError: Unexpected token -" | SyntaxError: Unexpected token -

Once runtime is fixed we can fix the BE compilation after.