feat(exporter-otlp-*)!: support custom HTTP agents #5719

raphael-theriault-swi · 2025-05-29T20:01:09Z

Which problem is this PR solving?

This makes it possible to use a custom http.Agent with the exporters, which enables some useful use cases, mainly proxying, without having to implement them in the SDK.

Fixes #5711

Short description of the changes

The httpAgentOptions option now also accepts an HTTP agent factory function. All internal code is migrated to use the factory; if an options object is provided, it is converted to a function. The function is only called once an agent is needed for export, which makes it possible to lazy import the http module and avoid issues where it is loaded before being instrumented.

The dynamic require of http-transport-utils has also been migrated to dynamic import, which should make it possible to bundle with Rollup. This works in tests by overriding the ts-node options so that the import call gets transpiled to require (only for tests) which loads fine.

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Added a test to make sure the custom agent is in fact being used.

Checklist:

Followed the style guidelines of this project
Unit tests have been added
Documentation has been updated

codecov · 2025-05-29T20:05:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.04%. Comparing base (88e8b2c) to head (129bcdf).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5719   +/-   ##
=======================================
  Coverage   95.04%   95.04%           
=======================================
  Files         307      307           
  Lines        7986     7993    +7     
  Branches     1614     1614           
=======================================
+ Hits         7590     7597    +7     
  Misses        396      396

Files with missing lines	Coverage Δ
.../configuration/convert-legacy-node-http-options.ts	`100.00% <100.00%> (ø)`
...ase/src/configuration/legacy-node-configuration.ts	`100.00% <ø> (ø)`
...-base/src/configuration/otlp-http-configuration.ts	`100.00% <100.00%> (ø)`
...packages/otlp-exporter-base/src/index-node-http.ts	`100.00% <100.00%> (ø)`
...rter-base/src/transport/http-exporter-transport.ts	`95.83% <100.00%> (+1.38%)`	⬆️
...xporter-base/src/transport/http-transport-utils.ts	`100.00% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts

raphael-theriault-swi · 2025-06-03T16:41:41Z

Updated to use a factory function for the agent, this iteration definitely feels cleaner than the previous one. I was also able to make what you attempted with #5220 work with the tests @pichlermarc (except now it breaks the instrumentation tests, looking into that)

pichlermarc · 2025-06-06T15:26:19Z

@raphael-theriault-swi nice 🙂

Before doing another round of reviews: it looks like the API tests are now failing here due to the changed settings:

opentelemetry-js/api/test/common/internal/version.test.ts

Lines 27 to 28 in a4e9645

    
           const pjson = require('../../../package.json'); 
        
           assert.strictEqual(pjson.version, VERSION);

- could you have a look?

experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts

raphael-theriault-swi · 2025-06-06T20:14:00Z

@pichlermarc I'm looking into it, the failure is not reproducible on macOS (with Node 20.6.0) so it'll have to wait a bit until next week when I have time to set up an environment where it also fails

experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts

raphael-theriault-swi · 2025-06-12T15:06:58Z

This PR is currently on hold as there's no way to make the tests work with ts-node on all the Node.js versions we test in CI (and believe me I tried). I'm working on a new PR to move to tsx as discussed during the 2025/06/11 SIG.

raphael-theriault-swi · 2025-06-20T18:47:45Z

@pichlermarc I've been banging my head for a week trying to migrate the tests to use something other than ts-node but it would require some pretty big changes since all of the module mocking with sinon kind of falls apart. I'll open an issue detailing my findings working on it in case anyone wants to pick up the work, but in the meantime I figured out how to make the tests work with this PR as-is.

raphael-theriault-swi · 2025-07-30T17:10:08Z

@pichlermarc would appreciate a review when you have some time :)

pichlermarc

Overall looks good already, thank you for working on this. 🙂

By looking at the way you solved the lazy-loading of the Agent I noticed that we could apply something similar to get rid of lazy-loading the full http-transport-util.ts file by just lazy-loading the request function instead.

That allows us to undo some of the changes to the mocharc and tsconfig files, leaving configs a bit cleaner and also gets rid of the awkward file lazy-loading that causes everyone trouble:

diff --git a/.mocharc.js b/.mocharc.js
deleted file mode 100644
index 11f4a7d8c..000000000
--- a/.mocharc.js
+++ /dev/null
@@ -1,3 +0,0 @@
-module.exports = {
-  require: `${__dirname}/ts-node.js`,
-};
diff --git a/.mocharc.yml b/.mocharc.yml
new file mode 100644
index 000000000..45bd4bdc1
--- /dev/null
+++ b/.mocharc.yml
@@ -0,0 +1 @@
+require: 'ts-node/register'
diff --git a/experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts b/experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts
index 05d45d6dc..c75d0cd7d 100644
--- a/experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts
+++ b/experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts
@@ -14,10 +14,7 @@
  * limitations under the License.
  */
 
-import type {
-  HttpRequestParameters,
-  sendWithHttp,
-} from './http-transport-types';
+import type { HttpRequestParameters } from './http-transport-types';
 
 // NOTE: do not change these type imports to actual imports. Doing so WILL break `@opentelemetry/instrumentation-http`,
 // as they'd be imported before the http/https modules can be wrapped.
@@ -25,10 +22,11 @@ import type * as https from 'https';
 import type * as http from 'http';
 import { ExportResponse } from '../export-response';
 import { IExporterTransport } from '../exporter-transport';
+import { sendWithHttp } from './http-transport-utils';
 
 interface Utils {
   agent: http.Agent | https.Agent;
-  send: sendWithHttp;
+  request: typeof http.request | typeof https.request;
 }
 
 class HttpExporterTransport implements IExporterTransport {
@@ -37,10 +35,11 @@ class HttpExporterTransport implements IExporterTransport {
   constructor(private _parameters: HttpRequestParameters) {}
 
   async send(data: Uint8Array, timeoutMillis: number): Promise<ExportResponse> {
-    const { agent, send } = await this._loadUtils();
+    const { agent, request } = await this._loadUtils();
 
     return new Promise<ExportResponse>(resolve => {
-      send(
+      sendWithHttp(
+        request,
         this._parameters,
         agent,
         data,
@@ -60,16 +59,10 @@ class HttpExporterTransport implements IExporterTransport {
     let utils = this._utils;
 
     if (utils === null) {
-      // Lazy import to ensure that http/https is not required before instrumentations can wrap it.
-      const imported = await import('./http-transport-utils.js');
-
+      const protocol = new URL(this._parameters.url).protocol;
       utils = this._utils = {
-        agent: await this._parameters.agent(
-          new URL(this._parameters.url).protocol
-        ),
-        // @ts-expect-error dynamic imports are never transpiled, but named exports only work when
-        // dynamically importing an esm file, and the utils file might be transpiled to cjs
-        send: imported.sendWithHttp ?? imported.default.sendWithHttp,
+        agent: await this._parameters.agent(protocol),
+        request: await getRequestFunction(protocol),
       };
     }
 
@@ -77,6 +70,18 @@ class HttpExporterTransport implements IExporterTransport {
   }
 }
 
+async function getRequestFunction(
+  protocol: string
+): Promise<typeof http.request | typeof https.request> {
+  if (protocol === 'http:') {
+    const { request } = await import('http');
+    return request;
+  } else {
+    const { request } = await import('https');
+    return request;
+  }
+}
+
 export function createHttpExporterTransport(
   parameters: HttpRequestParameters
 ): IExporterTransport {
diff --git a/experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts b/experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts
index 5c9797bc9..11a982c1a 100644
--- a/experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts
+++ b/experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts
@@ -14,19 +14,8 @@
  * limitations under the License.
  */
 
-import type * as http from 'http';
-import type * as https from 'https';
-import { ExportResponse } from '../export-response';
 import { HttpAgentFactory } from '../configuration/otlp-http-configuration';
 
-export type sendWithHttp = (
-  params: HttpRequestParameters,
-  agent: http.Agent | https.Agent,
-  data: Uint8Array,
-  onDone: (response: ExportResponse) => void,
-  timeoutMillis: number
-) => void;
-
 export interface HttpRequestParameters {
   url: string;
   headers: () => Record<string, string>;
diff --git a/experimental/packages/otlp-exporter-base/src/transport/http-transport-utils.ts b/experimental/packages/otlp-exporter-base/src/transport/http-transport-utils.ts
index 331d70757..5aa00e5eb 100644
--- a/experimental/packages/otlp-exporter-base/src/transport/http-transport-utils.ts
+++ b/experimental/packages/otlp-exporter-base/src/transport/http-transport-utils.ts
@@ -13,8 +13,8 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-import * as http from 'http';
-import * as https from 'https';
+import type * as http from 'http';
+import type * as https from 'https';
 import * as zlib from 'zlib';
 import { Readable } from 'stream';
 import { HttpRequestParameters } from './http-transport-types';
@@ -27,6 +27,7 @@ import { OTLPExporterError } from '../types';
 
 /**
  * Sends data using http
+ * @param requestFunction
  * @param params
  * @param agent
  * @param data
@@ -34,6 +35,7 @@ import { OTLPExporterError } from '../types';
  * @param timeoutMillis
  */
 export function sendWithHttp(
+  requestFunction: typeof https.request | typeof http.request,
   params: HttpRequestParameters,
   agent: http.Agent | https.Agent,
   data: Uint8Array,
@@ -53,9 +55,7 @@ export function sendWithHttp(
     agent: agent,
   };
 
-  const request = parsedUrl.protocol === 'http:' ? http.request : https.request;
-
-  const req = request(options, (res: http.IncomingMessage) => {
+  const req = requestFunction(options, (res: http.IncomingMessage) => {
     const responseData: Buffer[] = [];
     res.on('data', chunk => responseData.push(chunk));
 
diff --git a/experimental/packages/otlp-exporter-base/tsconfig.json b/experimental/packages/otlp-exporter-base/tsconfig.json
index 51b04951d..2bff90df9 100644
--- a/experimental/packages/otlp-exporter-base/tsconfig.json
+++ b/experimental/packages/otlp-exporter-base/tsconfig.json
@@ -19,12 +19,5 @@
     {
       "path": "../otlp-transformer"
     }
-  ],
-  "ts-node": {
-    "experimentalResolver": true,
-    "compilerOptions": {
-      "module": "commonjs",
-      "moduleResolution": "node10"
-    }
-  }
+  ]
 }
diff --git a/ts-node.js b/ts-node.js
deleted file mode 100644
index f1f97346e..000000000
--- a/ts-node.js
+++ /dev/null
@@ -1,8 +0,0 @@
-require('ts-node/register');
-
-if (process.env.MOCHA_PATCH_ESM != null) {
-  const { register } = require('module');
-  const { pathToFileURL } = require('url');
-
-  register('ts-node/esm', pathToFileURL(__filename));
-}

experimental/packages/otlp-exporter-base/src/configuration/convert-legacy-node-http-options.ts

Co-authored-by: Marc Pichler <[email protected]>

pichlermarc

Looks good to merge, just some nits around docs and such :)
Thanks for your patience :)

scripts/update-ts-configs.js

experimental/packages/otlp-exporter-base/src/configuration/legacy-node-configuration.ts

experimental/CHANGELOG.md

Co-authored-by: Marc Pichler <[email protected]>

raphael-theriault-swi · 2025-08-20T15:28:57Z

Updated the title at the same time for the breaking change

linux-foundation-easycla · 2025-08-20T17:02:22Z

The committers listed above are authorized under a signed CLA.

✅ login: raphael-theriault-swi / name: Raphaël Thériault (30edb48, 775606a, 597dace, 61d2e48, 215dcbd, d3c49f7, b7bad36, 137082d, d56f340, a554b70, c8a9930, ca15d41, 58f9655, 3166b11, e1fe39d)
✅ login: pichlermarc / name: Marc Pichler (129bcdf)

raphael-theriault-swi · 2025-08-20T17:07:38Z

Not sure what is happening here given the commits are in fact linked to my account

raphael-theriault-swi · 2025-08-20T18:07:30Z

I can try and rebase without the co-author comments given that's what seems to break it

trask · 2025-08-20T18:23:46Z

/easycla

trask · 2025-08-20T18:25:26Z

I can try and rebase without the co-author comments given that's what seems to break it

sorry folks, see open-telemetry/community#2920 (comment), we've rolled back the change for now, should be all good here

raphael-theriault-swi requested a review from a team as a code owner May 29, 2025 20:01

raphael-theriault-swi force-pushed the http-agent-instance branch from af65d9f to 7d4481b Compare May 29, 2025 20:01

pichlermarc reviewed Jun 2, 2025

View reviewed changes

experimental/packages/otlp-exporter-base/src/transport/http-transport-types.ts Outdated Show resolved Hide resolved

raphael-theriault-swi force-pushed the http-agent-instance branch from 59b612a to 06fa08b Compare June 3, 2025 16:49

raphael-theriault-swi mentioned this pull request Jun 6, 2025

fix(otlp-exporter-base): remove usage of require in esm/esnext builds #5746

Closed

2 tasks

raphael-theriault-swi added 6 commits June 6, 2025 12:40

feat(exporter-otlp-*): support custom http agents

597dace

chore(otlp-exporter-base): use an http agent factory

61d2e48

fix: don't patch esm loading in tests unconditionally

d56f340

fix(otlp-exporter-base): don't import http in common tests

775606a

chore(otlp-exporter-base): add legacy agent factory test

a554b70

chore: opt out switch for esm patching in test runner

58f9655

raphael-theriault-swi force-pushed the http-agent-instance branch from 5ee1842 to 58f9655 Compare June 6, 2025 16:41

EmrysMyrddin reviewed Jun 6, 2025

View reviewed changes

experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts Outdated Show resolved Hide resolved

fix: don't patch esm loading in http instrumentation test

b7bad36

fix: make mocha esm patching opt-in

ca15d41

JacksonWeber reviewed Jun 11, 2025

View reviewed changes

experimental/packages/otlp-exporter-base/src/transport/http-exporter-transport.ts Outdated Show resolved Hide resolved

david-luna mentioned this pull request Jun 16, 2025

[META 896] Set User-Agent in OTLP exporters elastic/elastic-otel-node#431

Open

3 tasks

fix(otlp-exporter-base): mess with ts-node so it works

3166b11

raphael-theriault-swi force-pushed the http-agent-instance branch from 7c9eae4 to 3166b11 Compare June 20, 2025 18:29

fix: keep ts-node section of tsconfig

30edb48

raphael-theriault-swi requested a review from pichlermarc June 20, 2025 18:44

dyladan mentioned this pull request Aug 6, 2025

OTLP-Exporter Proxy settings set are ignored #5835

Closed

Merge branch 'main' into http-agent-instance

d3c49f7

pichlermarc reviewed Aug 13, 2025

View reviewed changes

experimental/packages/otlp-exporter-base/src/configuration/convert-legacy-node-http-options.ts Outdated Show resolved Hide resolved

raphael-theriault-swi and others added 2 commits August 14, 2025 17:31

Merge branch 'main' into http-agent-instance

215dcbd

apply review suggestions

e1fe39d

Co-authored-by: Marc Pichler <[email protected]>

raphael-theriault-swi requested a review from pichlermarc August 14, 2025 22:11

fix: missing pr tidbits

c8a9930

pichlermarc approved these changes Aug 18, 2025

View reviewed changes

scripts/update-ts-configs.js Outdated Show resolved Hide resolved

experimental/packages/otlp-exporter-base/src/configuration/legacy-node-configuration.ts Show resolved Hide resolved

experimental/CHANGELOG.md Outdated Show resolved Hide resolved

raphael-theriault-swi changed the title ~~feat(exporter-otlp-*): support custom HTTP agents~~ feat(exporter-otlp-*)!: support custom HTTP agents Aug 20, 2025

apply review suggestions

137082d

Co-authored-by: Marc Pichler <[email protected]>

pichlermarc added this pull request to the merge queue Aug 20, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 20, 2025

pichlermarc added this pull request to the merge queue Aug 20, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 20, 2025

Merge branch 'main' into http-agent-instance

129bcdf

This comment was marked as resolved.

Sign in to view

pichlermarc added this pull request to the merge queue Aug 20, 2025

Merged via the queue into open-telemetry:main with commit d627fff Aug 20, 2025
24 checks passed

feat(exporter-otlp-*)!: support custom HTTP agents #5719

feat(exporter-otlp-*)!: support custom HTTP agents #5719

Uh oh!

Conversation

raphael-theriault-swi commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which problem is this PR solving?

Short description of the changes

Type of change

How Has This Been Tested?

Checklist:

Uh oh!

codecov bot commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

raphael-theriault-swi commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pichlermarc commented Jun 6, 2025

Uh oh!

Uh oh!

raphael-theriault-swi commented Jun 6, 2025

Uh oh!

Uh oh!

raphael-theriault-swi commented Jun 12, 2025

Uh oh!

raphael-theriault-swi commented Jun 20, 2025

Uh oh!

raphael-theriault-swi commented Jul 30, 2025

Uh oh!

pichlermarc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pichlermarc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raphael-theriault-swi commented Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raphael-theriault-swi commented Aug 20, 2025

Uh oh!

This comment was marked as resolved.

raphael-theriault-swi commented Aug 20, 2025

Uh oh!

trask commented Aug 20, 2025

Uh oh!

trask commented Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

raphael-theriault-swi commented May 29, 2025 •

edited

Loading

codecov bot commented May 29, 2025 •

edited

Loading

raphael-theriault-swi commented Jun 3, 2025 •

edited

Loading

linux-foundation-easycla bot commented Aug 20, 2025 •

edited

Loading