Segfault in 16.3.0 with promise hooks

[stephenh]

• Version: v16.3.0
• Platform:Linux sh12 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:22:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: async_hooks

What steps will reproduce the bug?

I cannot reproduce it outside of our internal codebase yet.

How often does it reproduce? Is there a required condition?

It deterministically reproduces in the test I am running, but I cannot reproduce it outside of that yet

What do you see instead?

A seg fault:

   1   │ PID 457702 received SIGSEGV for address: 0x0
   2   │ /home/stephen/homebound/graphql-service/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x3745)[0x7fde18334745]
   3   │ /lib/x86_64-linux-gnu/libpthread.so.0(+0x141f0)[0x7fde17f971f0]
   4   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node(_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+0x17)[0xd0cfc7]
   5   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node[0xa61a68]
   6   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node[0xd655cb]
   7   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node[0xd66a6c]
   8   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node(_ZN2v88internal21Builtin_HandleApiCallEiPmPNS0_7IsolateE+0x16)[0xd670e6]
   9   │ /home/stephen/.asdf/installs/nodejs/16.3.0/bin/node[0x16148d9]

Additional information

cc'ing @Qard because of the SetPromiseHooks in the above segfault.

Sorry I don't have any useful reproductions yet :-( will keep poking at it.

[gireeshpunathil]

@stephenh - do you have a core dump produced out of the crash that I can have a look at?

[stephenh]

core.zip

@gireeshpunathil yep, attached!

[gireeshpunathil]

(gdb) x/12i 0xD0CFB0
   0xd0cfb0 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_>:	push   rbp
   0xd0cfb1 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+1>:	mov    rbp,rsp
   0xd0cfb4 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+4>:	push   r15
   0xd0cfb6 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+6>:	push   r14
   0xd0cfb8 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+8>:	push   r13
   0xd0cfba <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+10>:	push   r12
   0xd0cfbc <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+12>:	mov    r12,rdi
   0xd0cfbf <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+15>:	push   rbx
   0xd0cfc0 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+16>:	mov    rbx,r8
   0xd0cfc3 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+19>:	sub    rsp,0x18
=> 0xd0cfc7 <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+23>:	mov    rax,QWORD PTR [rdi]
   0xd0cfca <_ZN2v87Context15SetPromiseHooksENS_5LocalINS_8FunctionEEES3_S3_S3_+26>:	and    rax,0xfffffffffffc0000
(gdb) i r rdi
rdi            0x0                 0
(gdb) 

basically, the Context is null. Not sure how this scenario is possible.

/cc @nodejs/v8

[Flarna]

There are known issues regarding PromiseHooks in 16.2.0 and 16.3.0. Latest fix was #38912

Above fix is already in master but not on 16.x yet. Could you test with a nightly build?

[stephenh]

@Flarna sure, I just tried a nightly, assuming I did it right, and still got a core dump:

I have the 6/13 nightly:

$ npx node-nightly --version
v17.0.0-nightly20210613174b191f70

Running this test segfaults:

$ npx node-nightly ./node_modules/.bin/jest src/...sometest....test.tsx

 RUNS  src/...sometest...test.tsx

I got this core dump file:

$ ls /tmp/core-node-11-1000-1000-1111527-1623592549 -l
-rw------- 1 stephen stephen 405385216 Jun 13 08:55 /tmp/core-node-11-1000-1000-1111527-1623592549

Which I'm attaching (...er, I'm getting a github error about "not in the list", but I'm uploading a zip file which worked before...still poking at it...)

[stephenh]

I can't seem to upload either a zip or a tar.gz file, and the error looks kinda weird/not display correctedly, in both firefox and chrome:

Not sure how I was able to upload the first one, it just worked...

[gireeshpunathil]

may be that this issue is already loaded with a heavy core file, and github is capping on size limit?

[stephenh]

Okay, I've put the core dump in this other repo (which was for reproducing a prior async hooks issue, the one in 16.2.0, not this current issue, I'm just borrowing that repository as a place to shove this file):

https://github.com/stephenh/async-local-storage-repro/blob/main/core-nightly-0613.tar.gz

[Qard]

This is the corresponding function. I don't see anything jumping out at me about it. 🤔 https://github.com/v8/v8/blob/master/src/api/api.cc#L6314-L6351

[Qard]

I suspect the issue may have been introduced by #38821. I think it's trying to do a context->SetPromiseHooks(...) when the context is not entered which may result in context->native_context() returning null.

[gireeshpunathil]

I can attempt a bisect. @stephenh, the reproduce repo that is linked - it has different steps and results, but I could not locate the one that produce the above crash. Can you guide me how do I reproduce the crash?

[stephenh]

@gireeshpunathil sorry, I don't have a repro for this one yet :-/

That repository was for reproducing #38821, and I just borrowed it for this issue's coredump.

For #38821, it was failing ~most/all of our tests, so it is was fairly easy to scope down to a few lines of code. This issue fails maybe ~10% of our tests (but fails those tests deterministically, which is at least good), but I haven't had a chance to dig into why.

At first I thought it was b/c of a particularly complex test, but later I noticed it is also segfaulting what I thought was a relatively simple test (we use AsyncLocalStorage within an ORM and the tests are using that + making a handful of SQL calls to a pg database). Which should be easier to pare down, but unfortunately still haven't had a chance to do so.

(...erg, last time we were also able to reproduce this in the ORM's public test suite, but it looks like the public test suite is passing on node 16.3.0, so it's only our internal application that is failing here...)