Build ID Section for WASM

[mitsuhiko]

I originally brought this up in the design repo (WebAssembly/design#1306) but I believe this fits here better.

For deferred symbolication on services like sentry it would be nice to be able to match up DWARF debug information to the main WASM file by build ID. In ELF this is typically accomplished with the GNU build ID note, on windows with the PDB signature and age and on darwin the macho UUID fulfills that purpose.

I would love to see a build_id custom section that contains a 16 or 20 byte ID which tools would ensure remains in both WASM files (CODE, debug companion containing DWARF info) if they get split. Capping it at 16 bytes makes it possible to roundtrip this through breakpad which uses a 16+4 byte char array for the debug id. 16 for the PDB UUID + 4 byte for the PDB age.

Motivation: Sentry and other systems like to be able to look up files by build ID because then they can access an external symbol server for that information. That way one just provides some sources where debug information can be found and then symbolicators just reach out to that service to find the debug information files.

[kripken]

Sounds good to me.

Would the build ID be modified by tools that process the wasm, say by Binaryen when it optimizes the binary? Or would it stay fixed after it's emitted from the original compiler? (The former seems to make sense, as optimizations change the binary, but then we'd need to describe those changes here I think.)

[mitsuhiko]

Definitely should change as the file changes.

Of note is that in the Microsoft ecosystem the age on the PDB signature (those extra 4 bytes) get incremented with every transformation. This from my experience has made things more complicated in practice because they were not consistently changed everywhere. For instance the age is stored more than once in the PE format and actually comes out desynched from Microsoft's own tools.

I think it would be wiser to explicitly tell tools to always completely override the embedded ID if it goes through a transformation. This does mean you can't track back to the original ID of the originally created WASM file but I'm not sure if that is necessary in general.

Would be curious to hear though if there are some advantages of the pdb+age system on the Microsoft side.

[sunfishcode]

How important is it to have an explicit field for this, as opposed to just having tools compute a hash of a wasm binary to use as an effective build ID?

[mitsuhiko]

@sunfishcode since the ID needs to survive a stripping of the file, it's very important. With DWARF in place you normally want to separate out the object file into two: one that contains CODE and other sections necessary to run the code, a second one with the DWARF sections (.debug_frame etc.). Since stripping/splitting the files changes the file you cannot reproduce the ID after the split.

[sunfishcode]

Can tools just hash the contents of the main wasm sections then, and ignore debug info sections?

I don't have a strong opinion either way yet; I just want to understand the space.

[mitsuhiko]

To generate the build ID they could take the hash of the main wasm sections and store it in the file. They can alternatively just generate a random UUID and embed it. I do think though that a build ID should ideally always be embedded.

(This here describes the workflow where this information is particularly useful)

[sunfishcode]

I'm curious about what situations storing a Build ID in the file is better than computing a hash on demand whenever it's needed.

Naively, computing it on demand would seem to have several advantages:

• tools can pick an appropriately strong hash function for their own use case
• tools can decide which sections to hash and which to ignore for their own use case
• tools don't have to worry about other tools stripping or omitting the Build ID
• tools can trust that no other tools have modified the binary without updating the Build ID
• tools can trust that the Build ID is not maliciously crafted to cause collisions, if that's important
• tools don't have to worry about the length of the Build ID varying

[Jake-Shadle]

FWIW Breakpad supports the embedded, format specific identifiers, that @mitsuhiko mentioned, but if they aren't available for any reason, it falls back to computing an md5 of the first 1024 bytes of the TEXT section (or equivalent).

You bring up some good points about some advantages computing the build ID has, but to me the point of the Build ID is to precisely identify a particular build so that different tools can always pair the code with the debug information, so allowing tools to choose their own hash function or which sections to hash, brings up problems when tools need to communicate with each other, eg. between a debugger and a symbol store that use different hash functions.

So storing the Build ID does have some disadvantages, particularly when a tool does a transformation that doesn't also change the Build ID, but I'm much more concerned with tools having a consistent source of truth.

[sunfishcode]

I've now found this stackoverflow post which I found helpful. The Build ID isn't just a hash of the contents; it's something like a hash of the contents and the debug info together, which is then recorded and preserved, even if debug info is stripped. As such, it can't always be recomputed.

There are a lot of use cases other than debug info that would seem to want something like a Build ID, but what they need is something subtly different from what the Build ID actually is. So, brainstorming here, what if we do have a Build ID section, but call it the "Debug Info ID", and say:

• Compilers producing wasm without debug info don't need to include a Debug Info ID.
• Post-processing tools could decide whether to strip the Debug Info ID based on whether their transformation would invalidate any associated debug info.

Would that make sense?

[mitsuhiko]

I think it's fair to specifically call this a debug_id and record that it's useful for that purpose.

The hashing fallback path of breakpad has caused more issues than it solved so I would prefer we don't spec out something like this.

[sbc100]

Would this build ID be generated at the point when the debug info is split out (either by the linker, or some kind of post link debug-splitting tool)? Or would it be present even in binaries that still have their debug info embedded?

[mitsuhiko]

@sbc100 definitely already in binaries that have the debug info embedded. We for instance have lots of cases where we want to symbolicate stacktraces where the client just submitted instruction addresses and then people upload the entire binary with debug information included.

This is especially important normally when doing stack unwinding out of memory dumps. This obviously is less useful for wasm right now, but in terms of existing work flows having the debug ID even in unstripped binaries has been very valuable.

[RReverser]

Just as a counter-point, one downside of an embedded id seems to be precisely that it would usually survive destructive operations on the code.

That is, if code is post-processed by a tool similar to wasm-opt or wasm-bindgen, and if that tool can't correctly update DWARF information, then the build id would remain the same even though the code has changed and no longer matches the debug info. In this case you as a consumer (Sentry or otherwise) explicitly don't want such debug info to be matched and used.

Arguably, every such tool should either support DWARF or be able to at least change build ID to some new unique value, but it seems that hashing of code section would alleviate this concern even more naturally.