Attach and debug any child process fork'ed by the target process

[derekparker]

Currently Delve does not use PTRACE_O_TRACEFORK or any other mechanism to begin debugging a process created when the target process (the original process being debugged) forks. This is an issue for tracking and discussing this new potential feature.

This would require changes to the API and to the debugger layer to allow for controlling and manipulating multiple processes simultaneously. It also requires some thought on how such a situation should behave from a users perspective.

Open questions / items to consider:

• Amount of work needed to make the debugger layer be able to handle multiple processes. We abstract a lot away via the Target type so the debugger would essentially need to initialize and keep track of N different child Targets.
• What new functionality will be needed from an API perspective? We would obviously need a way to switch between which is the "current" process which would affect any command used to manipulate or explore a process being debugged.
• How do we coordinate the two processes? Stop both when one hits a breakpoint or only the process which hit it? What happens when both programs hit a breakpoint, which do we show / prioritize for the user?

There's certainly a lot more to consider here but this is at least a brief list to get the topic started.

[mdempsky]

Something else not explicitly stated is that the subprocess might exec, so ideally delve would be able to debug multiple processes running different executables, including setting break points in processes+programs that aren't yet running. (Maybe this is implicitly understood as a requirement though.)

As a concrete use case to consider: I work on the Go compiler, and often I'm running something like go build std cmd or go run run.go (where run.go runs all of the tests in $GOROOT/test). Usually I care about some individual compile process deeply nested within this process tree. A limiting factor in my use of Delve is that I need to first reconstruct the exact environment and command-line that was used to invoke the failing compile process, and then use that to invoke Delve. This is a large hassle currently, so I haven't attempted it many times.

If I could just run the entire cmd/go process under Delve, set a breakpoint somewhere in cmd/compile, and then let delve handle all the details in letting me debug from there, that would be great.

--

As for implementation details, for reference, gdb supports multi-process debugging with "follow-forks-mode": https://sourceware.org/gdb/onlinedocs/gdb/Forks.html

Also relevant: https://sourceware.org/gdb/onlinedocs/gdb/Inferiors-Connections-and-Programs.html

I don't have much experience using it though. (I wanted to use it to debug the Chrome Linux sandbox, but Linux kernel bugs precluded that at the time.) I seem to recall you had the option to either allow the processes to each run independently or to have stop-the-world style breakpoints (which I assume were gdb's responsibility to pause all of the other processes, and thus inherently somewhat race-y).

[aarzilli]

The minimal implementation of this is to give the option to the user to stop the process on every exec (or maybe every exec that matches a given regexp) so that the user can, on a different window, run dlv attach on the new process. This is simple enough and would require almost no UI or API changes. But it would not be possible to do things like having delve stop all the processes.

[stapelberg]

so ideally delve would be able to debug multiple processes running different executables

Just to also mention it explicitly: there are also use-cases where we would like to use delve to debug multiple processes running the same executable.

Specifically, some test environments require re-executing the program after changing the environment (e.g. sandboxing), and that currently precludes using delve.

Sometimes, it’s not the environment, but the test itself, that requires re-execution. https://golang.org/src/os/exec/exec_test.go is such an example.

Thanks for considering this feature request :)

[myitcv]

Just to echo @stapelberg's example, my use case is testscript which re-executes the test executable in the course of running a test script to run commands registered by the test setup.

[katiehockman]

This is also going to be useful when debugging the fuzzing engine for Go native fuzzing. When fuzzing is run with go test -fuzz=<target_name> there are several processes that are spun up. If we want to debug the fuzzing engine to look into something in the standard library, any one of those processes could be doing something interesting that we want to debug. I've been trying to debug something under the hood for fuzzing today, and was unable to.