AWS Lambda Instrumentation causes lambda failures when using streaming response

[thpierce]

What version of OpenTelemetry are you using?

1.9.0 (api), 2.0.0 (core)

What version of Node are you using?

Node.js 22.x

What did you do?

• Create new Lambda function using Node.js 22.x with function body:

export const handler = awslambda.streamifyResponse(
  async (event, responseStream, _context) => {
    console.log(event)
    console.log(responseStream)
    console.log(_context)
    responseStream.write('test payload');
    responseStream.end();
  }
)

• Test above with helloworld - Success (see success.log)
• Checkout https://github.com/open-telemetry/opentelemetry-lambda
• from nodejs folder npm install & npm run build
• Create lambda layer from resultant zip.
• Add lambda layer to function, set env var: AWS_LAMBDA_EXEC_WRAPPER /opt/otel-handler
• Test again with helloworld - Failure (see failure.log):
• TypeError: responseStream.write is not a function

What did you expect to see?

Expected instrumentation to work.

What did you see instead?

Instrumentation caused my lambda to fail.

Additional context

The problem appears to be opentelemetry-instrumentation-aws-lambda. The instrumentation basically looks for any handler and replaces it with a patched handler that has 3 args: event, context, and callback. However, with streaming response lambdas the handler is streamifyReponse which needs 4 args: event, context, responseStream, and callback. So, responseStream is overwritten with callback and becomes unusable.

Two solutions:

1. [Short term] Do not instrument if handler is a streamifyReponse - Callout, this means streaming response lambdas will not be supported
2. [Long term] Properly instrument streamifyResponse, fully supporting both "normal" and streaming response lambdas.

Let me know if I should cut a separate issue for 2, above.

[pichlermarc]

cc @jj22ee (component owner)

[trentm]

@thpierce It might also help to ask in the #otel-faas channel on the CNCF Slack (https://slack.cncf.io/). I know that Serkan has been active working on the OTel Lambda layer for Node.js and other languages there.

[jj22ee]

Regarding Solution 1:

We'll need a mechanism is disable the Lambda instrumentation. The current question is how to differentiate between regular Lambda handlers (const handler = async (event) => { ... }) and streamify handlers (const handler = awslambda.streamifyResponse( ... )).

Seeing how Lambda provides different inputs into the Lambda function solely based on having handler = awslambda.streamifyResponse means there's something special about this awslambda.streamifyResponse function.

Digging around I found that what makes it special is that it contains a special "Symbol" property:

• handler[Symbol.for("aws.lambda.runtime.handler.streaming")] = "response".

In this example I tried, you can trick Lambda into "enable streaming" for this regular Lambda by adding this property:

// handler will error out but still print out the logs to console
export const handler = async (event, context, callback, arg4) => {
  console.log(event)
  console.log(context)    // This now becomes a "responseStream" instead of context
  console.log(callback)   // This becomes a "context" instead of callback function
  console.log(arg4)
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  };
  return response;
};

// Make Lambda treat this normal handler as an "awslambda.streamifyResponse" handler
handler[Symbol.for("aws.lambda.runtime.handler.streaming")] = "response"

Conversely, you can trick Lambda into thinking that this "streaming enabled" Lambda is a normal handler by deleting that property:

export const handler = awslambda.streamifyResponse(
  async (event, responseStream, _context, d) => {
    console.log(event)
    console.log(responseStream)    // This now becomes a "context" instead of responseStream
    console.log(_context)          // This now becomes a "callback function" instead of a context
    console.log(d)
    // console.log(process.env)
    // console.log(handler)
    // responseStream.write('test payload');    // Won't work after deleting the Symbol property
    // responseStream.end();
  }
)

// Make Lambda treat this "awslambda.streamifyResponse" handler as a normal handler
delete handler[Symbol.for("aws.lambda.runtime.handler.streaming")]

That said, we can probably disable the Lambda instrumentation by not patching the Lambda handler here if the original handler contains this property, and just continue to use the original handler.

Regarding Solution 2:

Given what we know now, we could (probably) extend the current Lambda Instrumentation to better support patching awslambda.streamifyResponse by preserving the Symbol.for("aws.lambda.runtime.handler.streaming") property. This also means undoing the proposed Solution 1.

There might be more to this though, there are other possible properties that may also need to be preserved from the original handler like Symbol.for("aws.lambda.runtime.handler.streaming.highWaterMark"), which I don't know what it is for at this moment.