Datadog Receiver spans don't follow OTel specs

[cyrille-leclerc]

Component(s)

receiver/datadog

What happened?

Description

Spans emitted by Datadog APM agents and converted by the OTel Collector Datadog Receiver don't comply with OTel Semantic conventions, creating challenges to visualize them.

Misalignments applying to all span types

• A resource attribute deployment.resource.name should be defined instead of the span attribute deployment.environment: Str(production)
• A resource attribute process.pid should be defined instead of the attribute process.id

Misalignments applying to HTTP server span

Specs: https://opentelemetry.io/docs/specs/semconv/http/http-spans/

Example Datadog HTTP Server span produced by a Spring Boot app (code) and converted by the OTel Col Datadog receiver

2024-12-23T11:28:04.619+0100	info	ResourceSpans #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(shipping)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #0
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 288018d171bc9c67
    ID             : 3e65de74bd03a50a
    Name           : servlet.request
    Kind           : Server
    Start time     : 2024-12-23 10:28:03.717953709 +0000 UTC
    End time       : 2024-12-23 10:28:03.778773334 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(POST /shipOrder)
     -> sampling.priority: Str(1.000000)
     -> datadog.span.id: Str(4496244395968013578)
     -> datadog.trace.id: Str(17149159500458815648)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> http.status_code: Str(200)
     -> language: Str(jvm)
     -> component: Str(tomcat-server)
     -> deployment.environment: Str(production)
     -> peer.ipv4: Str(127.0.0.1)
     -> span.kind: Str(server)
     -> http.useragent: Str(Java-http-client/17.0.13)
     -> _dd.tracer_host: Str(cyrille-le-clerc-macbook.local)
     -> servlet.path: Str(/shipOrder)
     -> http.method: Str(POST)
     -> servlet.context: Str(/)
     -> http.route: Str(/shipOrder)
     -> runtime-id: Str(9136fc59-3ff8-4bf1-8ec6-725b8c6b8ef3)
     -> _dd.p.dm: Str(-0)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> http.url: Str(http://shipping.local:8088/shipOrder)
     -> service.version: Str(1.1)
     -> http.hostname: Str(shipping.local)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)
     -> process.pid: Double(43095)
     -> _dd.profiling.enabled: Double(0)
     -> peer.port: Double(59969)
     -> _dd.trace_span_attribute_schema: Double(0)
     -> _sampling_priority_v1: Double(1)
     -> _dd.measured: Double(1)

• Adopt new OTel Semantic Conventions http.request.method and http.response.status_code instead of http.method and http.status_code
• The span name should be ${http.request.method} ${http.route}, the attribute dd.span.Resource is right for this span type
  • OTTL transformation to fix the pb: set (name, attributes["dd.span.Resource"]) where name == "servlet.request"

Misalignment applying to internal Spring Framework spans(span name spring.handler)

This probably applies to other internal spans.

Example Datadog internal span produced by a Spring Boot app (code) and converted by the OTel Col Datadog receiver

Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(shipping)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #1
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 3e65de74bd03a50a
    ID             : 6a0035efa46106ae
    Name           : spring.handler
    Kind           : Server
    Start time     : 2024-12-23 10:28:03.720530834 +0000 UTC
    End time       : 2024-12-23 10:28:03.777623 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(ShippingController.shipOrder)
     -> datadog.span.id: Str(7638164271391639214)
     -> datadog.trace.id: Str(17149159500458815648)
     -> service.version: Str(1.1)
     -> span.kind: Str(server)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> language: Str(jvm)
     -> component: Str(spring-web-controller)
     -> deployment.environment: Str(production)
     -> _dd.measured: Double(1)
     -> thread.id: Double(38)

• Span should be kind = SpanKind.SPAN_KIND_INTERNAL
  • OTTL transformation to fix the pb: set (kind, SPAN_KIND_INTERNAL) where kind == SPAN_KIND_SERVER and name == "spring.handler"
• Span name should be the Java method nam hat is carried over by the dd.span.Resource attribute
  • OTTL transformation to fix the pb: `set (name, attributes["dd.span.Resource"]) where name

Misalignments applying to Database client spans

Specs: https://opentelemetry.io/docs/specs/semconv/database/database-spans/

Example Datadog DB client span produced by a Spring Boot app (code and converted by the OTel Col Datadog receiver

ResourceSpans #1
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> service.version: Str(1.1)
     -> telemetry.sdk.version: Str(Datadog-1.44.1~13a9a2d011)
     -> telemetry.sdk.name: Str(Datadog)
     -> telemetry.sdk.language: Str(java)
     -> process.runtime.version: Str(17.0.13)
     -> service.name: Str(postgresql)
     -> host.name: Str(localhost)
     -> os.type: Str(darwin)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope Datadog 1.44.1~13a9a2d011
Span #0
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 6a0035efa46106ae
    ID             : 2dc4a71b6d1cac05
    Name           : postgresql.query
    Kind           : Client
    Start time     : 2024-12-23 10:28:03.723354292 +0000 UTC
    End time       : 2024-12-23 10:28:03.775606 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(select pg_sleep(?))
     -> datadog.span.id: Str(3297944563378596869)
     -> datadog.trace.id: Str(17149159500458815648)
     -> db.type: Str(postgresql)
     -> db.user: Str(my_shopping_cart)
     -> deployment.environment: Str(production)
     -> component: Str(java-jdbc-statement)
     -> span.kind: Str(client)
     -> _dd.base_service: Str(shipping)
     -> db.operation: Str(select)
     -> peer.hostname: Str(postgresql.local)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> db.instance: Str(my_shopping_cart)
     -> db.pool.name: Str(HikariPool-1)
     -> _dd.measured: Double(1)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)
Span #1
    Trace ID       : 0000000000000000edfe0db49d5cf0a0
    Parent ID      : 6a0035efa46106ae
    ID             : 504c8390c5d6dc9b
    Name           : postgresql.query
    Kind           : Client
    Start time     : 2024-12-23 10:28:03.72148425 +0000 UTC
    End time       : 2024-12-23 10:28:03.723227917 +0000 UTC
    Status code    : Ok
    Status message :
Attributes:
     -> dd.span.Resource: Str(postgresql.query)
     -> sampling.priority: Str(1.000000)
     -> datadog.span.id: Str(5786144279081966747)
     -> datadog.trace.id: Str(17149159500458815648)
     -> span.kind: Str(client)
     -> _dd.base_service: Str(shipping)
     -> deployment.environment: Str(production)
     -> component: Str(java-jdbc-statement)
     -> db.type: Str(postgresql)
     -> db.instance: Str(my_shopping_cart)
     -> db.user: Str(my_shopping_cart)
     -> peer.hostname: Str(postgresql.local)
     -> db.pool.name: Str(HikariPool-1)
     -> _dd.p.tid: Str(2b129ecc5fb2e048)
     -> thread.name: Str(http-nio-8088-exec-2)
     -> _sampling_priority_v1: Double(1)
     -> _dd.measured: Double(1)
     -> _dd.top_level: Double(1)
     -> thread.id: Double(38)

• The resource attribute service.name should be set to the value of the span attribute _dd.base_service
• Rename the following span attributes
  • db.type -> db.system
  • db.operation -> db.operation.name
  • db.instance -> db.name
• The span name should be set to ${db.operation.name} ${}

Steps to Reproduce

• Instrument a Spring Boot App similar to this example with the Datadog Java Agent v1.44.1, send the traces to and OTel Col Datadog Receiver using the DD_TRACE_AGENT_URL="http://localhost:8126", note that setting DD_REMOTE_CONFIGURATION_ENABLED=false seems to prevent some problems of traces not being exported:

  java \
       -javaagent:"$DATADOG_AGENT_JAR" \
       -Dserver.port=8088 \
       -jar target/shipping-1.1-SNAPSHOT.jar
  

• Setup an OTel Collector v0.116.0 with a Datadog Receiver and exporting traces to the console for debug

  receivers:
    datadog:
      endpoint: localhost:8126
      read_timeout: 60s
  exporters:
    debug:
      verbosity: detailed
  service:
    pipelines:
      traces:
        receivers: [datadog]
        processors: 
        exporters: [debug]

Expected Result

OTel-compliant spans reported in the Otel Col stdout

Actual Result

The non OTel-compliant spans described above

Collector version

v0.116.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  datadog:
    endpoint: localhost:8126
    read_timeout: 60s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [datadog]
      processors: 
      exporters: [debug]

Log output

see issue description above

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/datadog: @boostchicken @gouthamve @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/datadog: @boostchicken @gouthamve @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

@boostchicken @gouthamve @MovieStoreGuy please review as codeowners.

@cyrille-leclerc would you like to help provide enhancements to change the behavior of the receiver?