OTLP Exporter does not respect retry configuration when backend is unavailable

[skhalash]

Description

When using the OpenTelemetry Go SDK with the OTLP gRPC exporter, we expect the client to retry failed connections using an exponential backoff strategy, as specified in the OTLP specification. However, in practice, this does not seem to be happening.

Environment

• OS: OSX
• Architecture: ARM
• Go Version: 1.24.1
• opentelemetry-go version: 1.35.0

Steps To Reproduce

Steps to Reproduce:

1. Run the following Go program with an OTLP gRPC exporter configured with retries. To simplify the example, we are exporting only spans (exporting metrics and logs has the same behavior):

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func main() {
	ctx := context.Background()

	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize trace exporter: %v", err)
	}

	tp, err := newTracerProvider(exp)
	if err != nil {
		log.Fatalf("failed to initialize trace provider: %v", err)
	}

	defer func() { _ = tp.Shutdown(ctx) }()

	otel.SetTracerProvider(tp)
	generateSpan()

	select {}
}

func generateSpan() {
	log.Println("Generating a dummy span")
	_, span := otel.Tracer("").Start(context.Background(), "dummy")
	defer span.End()
}

func newTracerProvider(exp sdktrace.SpanExporter) (*sdktrace.TracerProvider, error) {
	r, err := resource.Merge(
		resource.Default(),
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("ExampleService"),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create resource: %w", err)
	}

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(r),
	), nil
}

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	traceExporter, err := otlptrace.New(
		ctx,
		otlptracegrpc.NewClient(
			otlptracegrpc.WithEndpoint("127.0.0.1:4317"),
			otlptracegrpc.WithInsecure(),
			otlptracegrpc.WithRetry(otlptracegrpc.RetryConfig{
				Enabled:         true,
				InitialInterval: 1 * time.Second,
				MaxInterval:     30 * time.Second,
				MaxElapsedTime:  time.Minute,
			}),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create trace exporter: %w", err)
	}

	return traceExporter, nil
}

2. Ensure the OpenTelemetry Collector is NOT running.

3. Run the program and observe the logs.

4. Notice that there are no retries logged and it seem to give up after 15 seconds, even though there is a retry config specified:

2025/04/01 13:32:51 Generating a dummy span                                                                                                                                 
2025/04/01 13:33:06 traces export: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused"

The same behavior is observed with the otlphttp exporter as well as the other telemetry signals.

Expected behavior

If the OTLP backend is unavailable, the exporter should retry connecting based on the provided RetryConfig, implementing exponential backoff with jitter.

[yumosx]

HI, @skhalash , the withRetry's comment say:

WithRetry sets the retry policy for transient retryable errors that may be
returned by the target endpoint when exporting a batch of spans.

If the target endpoint responds with not only a retryable error, but
explicitly returns a backoff time in the response. That time will take
precedence over these settings.

These settings do not define any network retry strategy. That is entirely
handled by the gRPC ClientConn.

If unset, the default retry policy will be used. It will retry the export
5 seconds after receiving a retryable error and increase exponentially
after each error for no more than a total time of 1 minute

These settings do not define any network retry strategy. That is entirely
handled by the gRPC ClientConn.

I suspect this might be the reason.

[skhalash]

Hey @yumosx! Thanks for the answer. I understand that technically it should be handled by gRPC ClientConn, but according to my observations it's not handled at all, which contradicts the SDK spec guidelines:

If the client cannot connect to the server, the client SHOULD retry the connection using an exponential backoff strategy between retries. The interval between retries must have a random jitter.

Or am I misunderstanding something?

Is there any other way to configure retries in case of connection refusal? And it probably should be configurable for both otlphttp and otlpgrpc exporters.

[yumosx]

You're right. My mistake, this error can be retried.

[XSAM]

This is related to spec compliance.

It looks like we need to adjust the retry mechanism a little bit to meet the requirement.

[XSAM]

I found that the context deadline exceeded error is coming from other options, which prevents the retry mechanism from reaching the default MaxElapsedTime (1 minute). The retry mechanism is actually retrying when the backend is unavailable. This also means the exporter meets the spec compliance, but we need better documentation.

Timeout from BatchSpanProcessor

Default timeout: 30 seconds.

https://pkg.go.dev/go.opentelemetry.io/otel/sdk/trace#WithExportTimeout

opentelemetry-go/sdk/trace/batch_span_processor.go

Line 270 in 3ec70fd

ctx, cancel = context.WithTimeout(ctx, bsp.o.ExportTimeout)

This 30-second timeout is part of the spec https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#batch-span-processor

Timeout from OTLP gRPC Exporter itself

Default timeout: 10 seconds.

https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc#WithTimeout

opentelemetry-go/exporters/otlp/otlptrace/otlptracegrpc/client.go

Line 226 in 5ba5e7a

ctx, cancel = context.WithTimeout(parent, c.exportTimeout)

This 10-second timeout is also part of the spec https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_timeout

Solution

You can use trace.WithExportTimeout and otlptracegrpc.WithTimeout to overwrite the default timeout, so the retry mechanism can reach the MaxElapsedTime.

Example

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func main() {
	ctx := context.Background()

	exp, err := newExporter(ctx)
	if err != nil {
		log.Fatalf("failed to initialize trace exporter: %v", err)
	}

	tp, err := newTracerProvider(exp)
	if err != nil {
		log.Fatalf("failed to initialize trace provider: %v", err)
	}

	defer func() { _ = tp.Shutdown(ctx) }()

	otel.SetTracerProvider(tp)
	generateSpan()

	select {}
}

func generateSpan() {
	log.Println("Generating a dummy span")
	_, span := otel.Tracer("").Start(context.Background(), "dummy")
	defer span.End()
}

func newTracerProvider(exp sdktrace.SpanExporter) (*sdktrace.TracerProvider, error) {
	r, err := resource.Merge(
		resource.Default(),
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName("ExampleService"),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create resource: %w", err)
	}

	return sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp,
			sdktrace.WithExportTimeout(2*time.Minute),
		),
		sdktrace.WithResource(r),
	), nil
}

func newExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	traceExporter, err := otlptrace.New(
		ctx,
		otlptracegrpc.NewClient(
			otlptracegrpc.WithEndpoint("127.0.0.1:4317"),
			otlptracegrpc.WithInsecure(),
			otlptracegrpc.WithTimeout(2*time.Minute),
			otlptracegrpc.WithRetry(otlptracegrpc.RetryConfig{
				Enabled:         true,
				InitialInterval: 1 * time.Second,
				MaxInterval:     30 * time.Second,
				MaxElapsedTime:  time.Minute,
			}),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create trace exporter: %w", err)
	}

	return traceExporter, nil
}

The output is

2025/04/15 19:23:23 Generating a dummy span
2025/04/15 19:24:35 traces export: max retry time elapsed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4317: connect: connection refused"

Action item

I will create a PR to change the comments of WithRetry, mentioning that the retry behavior is affected by the timeout mechanism. I will also make the exporter return the cause of the timeout as an error, so users can know where the timeout comes from.

• Use the cause of the context error in OTLP retry #6898
• Retry otlphttp when connection refused.
• Update WithRetry comments

[skhalash]

@XSAM Thanks a lot, it works perfectly :) I also figured out that you can configure this behavior using env variables OTEL_EXPORTER_OTLP_TIMEOUT and OTEL_BSP_EXPORT_TIMEOUT. However, those env variables only work with otlpgrpc despite of generic names. With otlphttp, I am getting a connection refused second after 5 seconds.

And one more question - do you think it might make sense to set those timeouts to greater values by default? I find it a bit weird that the default retry policy max timeout is set to 1 minute, but it will not be respected in case of a backend outage? And the user must know about those other timeout configurations and explicitly change them.

[XSAM]

However, those env variables only work with otlpgrpc despite of generic names.

OTEL_EXPORTER_OTLP_TIMEOUT should also work for OTLP HTTP exporter. Proof:

• opentelemetry-go/exporters/otlp/otlptrace/otlptracehttp/internal/otlpconfig/options_test.go

  Line 434 in 12c7d7e

  "OTEL_EXPORTER_OTLP_TIMEOUT": "15000",

And, OTEL_BSP_EXPORT_TIMEOUT is an environment variable for the batch span processor, not the exporter.

With otlphttp, I am getting a connection refused second after 5 seconds.

@skhalash Could you share your code snippet to reproduce this? It might be because some HTTP errors are not considered retryable.

Good catch. The retry does not handle this situation. I have included this in the action items.

do you think it might make sense to set those timeouts to greater values by default? I find it a bit weird that the default retry policy max timeout is set to 1 minute

The timeout values for the processor and exporter are coming from the Spec; unless we change the spec, we cannot change these default timeout values. It is worth mentioning that the spec does not define the max timeout for retry.

Even if we increase the timeout for retry, it does not help because the context timeout comes from processors or exporters.

I think this is a trade-off or config tuning that users can adjust for their needs. For maintainers, we can clarify the behavior in the docs, so it won't become an unexpected behavior for users.