Pipeline aborts all HTTP requests

[szmarczak]

• Version: >= 13.10.0
• Platform: all
• Subsystem: stream

This bug breaks hundreds of packages, including Got, Yarn and Renovate.

What steps will reproduce the bug?

const {PassThrough, pipeline} = require('stream');
const https = require('https');

const body = new PassThrough();
const request = https.request('https://example.com');

request.once('abort', () => {
	console.log('The request has been aborted');
});

pipeline(
	body,
	request,
	error => {
		console.log(`Pipeline errored: ${!!error}`);
	}
);

body.end();

How often does it reproduce? Is there a required condition?

Always.

What is the expected behavior?

Pipeline errored: false

What do you see instead?

Pipeline errored: false
+The request has been aborted

Additional information

This is the culprit: https://github.com/nodejs/node/pull/31940/files#diff-eefad5e8051f1634964a3847b691ff84R36

->

https://github.com/nxtedition/node/blob/5a55b1df97a1f93c5fb81bf758050d81524ae834/lib/internal/streams/destroy.js#L166

/cc @ronag

[ronag]

I'll take a look asap

[ronag]

Are you able to create a self contained test case? I tried the following on master but was unable to make it fail:

  const server = http.createServer((req, res) => {
    setTimeout(() => {
      req.write('ASDASD');
      req.end();
    }, 1e3);
  });

  server.listen(0, () => {
    const req = http.request({
      port: server.address().port
    });

    const body = new PassThrough();
    pipeline(
      body,
      req,
      common.mustCall((err) => {
        assert(!err);
        server.close();
      })
    );
    body.end();
  });

[szmarczak]

You've missed

request.once('abort', common.mustNotCall);

[ronag]

Ah, I think this is actually a problem with the http_client, I guess it shouldn't emit 'abort' if it has gracefully ended.

@mcollina

[szmarczak]

No, you misunderstood me. Let me explain...

[szmarczak]

In #31940 you introduced this change:

https://github.com/nodejs/node/pull/31940/files#diff-eefad5e8051f1634964a3847b691ff84R36

It calls destroyer(stream, error), which calls

request.abort() even though it should not.

[ronag]

even though it should not

Why do you think it should not?

[szmarczak]

pipeline is meant to sent data from one stream to another. It shouldn't destroy streams that have been successfully ended. I mean it should call request.abort() only if an error occurs.

[ronag]

pipeline is meant to sent data from one stream to another. It shouldn't destroy streams that have been successfully ended. I mean it should call request.abort() only if an error occurs.

I disagree with this. The current semantics of pipeline is to destroy all streams on completion. This is because not all streams properly call destroy on completion, and the idea of pipeline is to normalize this behavior.

I might be wrong here, and it might also be a bad idea. Why do you think it shouldn't be destroyed?

And if we don't call destroy how do we properly handle streams that do no properly destroy themselves? Changing this could break it in the other direction where we end up with leaking resources.

[ronag]

A quick fix to resolve the break is not have a special case for request and simply don't destroy it like the rest. But I don't think that is a longterm solution. Do we have a more fundamental problem?

[ronag]

@mcollina: I think we need your guidance here.

[ronag]

@szmarczak What is the fundamental breaking change here? That the request is aborted/destroyed or that 'abort' is emitted?

[szmarczak]

That the request is aborted.

[ronag]

That the request is aborted.

If it has completed. Why is this a problem? Aborting a completed request is basically a noop expect for the 'abort' event?

Is the problem here that only the writable side has completed when the pipeline callback is invoked and the readable side is still to be consumed?