Skip to content

[9.0] Use elastic apm connector to replace other apm components #7172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Mar 4, 2025
Merged

[9.0] Use elastic apm connector to replace other apm components #7172

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1c4f049
Use elastic apm connector to replace other apm components
lahsivjar Feb 20, 2025
72d0d2d
Remove elasticapm connector for metrics
lahsivjar Feb 28, 2025
9ab37a4
Bump to latest, update license and tidy
lahsivjar Mar 4, 2025
f42772f
Fix toolchain
lahsivjar Mar 4, 2025
39b1c4e
license headers and checks
lahsivjar Mar 4, 2025
e0b324e
Bump up go bugfix version
lahsivjar Mar 4, 2025
1754c02
make license
lahsivjar Mar 4, 2025
69685f2
Fix go modules
lahsivjar Mar 4, 2025
7145433
fix notice
lahsivjar Mar 4, 2025
23fd237
Remove deprecated exporter.MakeFactoryMap in favor of otelcol.MakeFac…
lahsivjar Mar 4, 2025
948371e
Fix TestComponentsCommand test
lahsivjar Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,673 changes: 1,595 additions & 1,078 deletions NOTICE.txt
View file
Open in desktop

Large diffs are not rendered by default.

3 changes: 1 addition & 2 deletions deploy/helm/edot-collector/kube-stack/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ The OpenTelemetry components deployed within the `Gateway` Deployment collectors

- [Elastic Trace processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elastictraceprocessor): The processor enriches traces with elastic specific requirements. It uses opentelemetry-lib to perform the actual enrichments.
- [Elastic Infra Metrics processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor): The Elastic Infra Metrics Processor is used to bridge the gap between OTEL and Elastic Infra Metrics.
- [LSM interval processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/lsmintervalprocessor): [Interval processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor) with db-backed persistence.
- [Signal to Metrics connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/signaltometricsconnector): Produces metrics from all signal types (traces, logs, or metrics).
- [Elastic APM connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/elasticapmconnector): The Elastic APM connector produces aggregated Elastic APM-specific metrics from all telemetry signals.

### Auto-instrumentation

Expand Down
278 changes: 5 additions & 273 deletions deploy/helm/edot-collector/kube-stack/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,251 +563,8 @@ collectors:
- context: metric
statement: route() where instrumentation_scope.name == "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver" or IsMatch(instrumentation_scope.name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/*")
pipelines: [metrics/infra/ecs, metrics/otel]
# [Signal To Metrics Connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/signaltometricsconnector)
signaltometrics: # Produces metrics from all signal types (traces, logs, or metrics).
logs:
- name: service_summary
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: metricset.name
default_value: service_summary
sum:
value: "1"
datapoints:
- name: service_summary
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: metricset.name
default_value: service_summary
sum:
value: "1"
spans:
- name: service_summary
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: metricset.name
default_value: service_summary
sum:
value: Int(AdjustedCount())
- name: transaction.duration.histogram
description: APM service transaction aggregated metrics as histogram
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: transaction.root
- key: transaction.type
- key: metricset.name
default_value: service_transaction
- key: elasticsearch.mapping.hints
default_value: [_doc_count]
unit: us
exponential_histogram:
value: Microseconds(end_time - start_time)
- name: transaction.duration.summary
description: APM service transaction aggregated metrics as summary
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: transaction.root
- key: transaction.type
- key: metricset.name
default_value: service_transaction
- key: elasticsearch.mapping.hints
default_value: [aggregate_metric_double]
unit: us
histogram:
buckets: [1]
value: Microseconds(end_time - start_time)
- name: transaction.duration.histogram
description: APM transaction aggregated metrics as histogram
ephemeral_resource_attribute: true
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
- key: container.id
- key: k8s.pod.name
- key: service.version
- key: service.instance.id # service.node.name
- key: process.runtime.name # service.runtime.name
- key: process.runtime.version # service.runtime.version
- key: telemetry.sdk.version # service.language.version??
- key: host.name
- key: os.type # host.os.platform
- key: faas.instance
- key: faas.name
- key: faas.version
- key: cloud.provider
- key: cloud.region
- key: cloud.availability_zone
- key: cloud.platform # cloud.servicename
- key: cloud.account.id
attributes:
- key: transaction.root
- key: transaction.name
- key: transaction.type
- key: transaction.result
- key: event.outcome
- key: metricset.name
default_value: transaction
- key: elasticsearch.mapping.hints
default_value: [_doc_count]
unit: us
exponential_histogram:
value: Microseconds(end_time - start_time)
- name: transaction.duration.summary
description: APM transaction aggregated metrics as summary
ephemeral_resource_attribute: true
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
- key: container.id
- key: k8s.pod.name
- key: service.version
- key: service.instance.id # service.node.name
- key: process.runtime.name # service.runtime.name
- key: process.runtime.version # service.runtime.version
- key: telemetry.sdk.version # service.language.version??
- key: host.name
- key: os.type # host.os.platform
- key: faas.instance
- key: faas.name
- key: faas.version
- key: cloud.provider
- key: cloud.region
- key: cloud.availability_zone
- key: cloud.platform # cloud.servicename
- key: cloud.account.id
attributes:
- key: transaction.root
- key: transaction.name
- key: transaction.type
- key: transaction.result
- key: event.outcome
- key: metricset.name
default_value: transaction
- key: elasticsearch.mapping.hints
default_value: [aggregate_metric_double]
unit: us
histogram:
buckets: [1]
value: Microseconds(end_time - start_time)
- name: span.destination.service.response_time.sum.us
description: APM span destination metrics
ephemeral_resource_attribute: true
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: span.name
- key: event.outcome
- key: service.target.type
- key: service.target.name
- key: span.destination.service.resource
- key: metricset.name
default_value: service_destination
unit: us
sum:
value: Double(Microseconds(end_time - start_time))
- name: span.destination.service.response_time.count
description: APM span destination metrics
ephemeral_resource_attribute: true
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: span.name
- key: event.outcome
- key: service.target.type
- key: service.target.name
- key: span.destination.service.resource
- key: metricset.name
default_value: service_destination
sum:
value: Int(AdjustedCount())
# event.success_count is populated using 2 metric definition with different conditions
# and value for the histogram bucket based on event outcome. Both metric definition
# are created using same name and attribute and will result in a single histogram.
# We use mapping hint of aggregate_metric_double, so, only the sum and the count
# values are required and the actual histogram bucket is ignored.
- name: event.success_count
description: Success count as a metric for service transaction
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: transaction.root
- key: transaction.type
- key: metricset.name
default_value: service_transaction
- key: elasticsearch.mapping.hints
default_value: [aggregate_metric_double]
conditions:
- attributes["event.outcome"] != nil and attributes["event.outcome"] == "success"
unit: us
histogram:
buckets: [1]
count: Int(AdjustedCount())
value: Int(AdjustedCount())
- name: event.success_count
description: Success count as a metric for service transaction
include_resource_attributes:
- key: service.name
- key: deployment.environment # service.environment
- key: telemetry.sdk.language # service.language.name
- key: agent.name
default_value: unknown
attributes:
- key: transaction.root
- key: transaction.type
- key: metricset.name
default_value: service_transaction
- key: elasticsearch.mapping.hints
default_value: [aggregate_metric_double]
conditions:
- attributes["event.outcome"] != nil and attributes["event.outcome"] != "success"
unit: us
histogram:
buckets: [0]
count: Int(AdjustedCount())
value: Double(0)
# [Elastic APM Connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/elasticapmconnector)
elasticapm: {}
receivers:
otlp:
protocols:
Expand Down Expand Up @@ -841,31 +598,8 @@ collectors:
# explicitly set send_batch_max_size to 0, as splitting metrics requests may cause version_conflict_engine_exception in TSDB
send_batch_max_size: 0
timeout: 1s
batch/aggs:
send_batch_size: 16384 # 2x the default
# explicitly set send_batch_max_size to 0, as splitting metrics requests may cause version_conflict_engine_exception in TSDB
send_batch_max_size: 0
timeout: 10s
# [Elastic Trace Processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elastictraceprocessor)
elastictrace: {} # The processor enriches traces with elastic specific requirements.
# [LSM Interval Processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/lsmintervalprocessor)
lsminterval:
intervals:
- duration: 1m
statements:
- set(resource.attributes["metricset.interval"], "1m")
- set(attributes["data_stream.dataset"], Concat([attributes["metricset.name"], "1m"], "."))
- set(attributes["processor.event"], "metric")
- duration: 10m
statements:
- set(resource.attributes["metricset.interval"], "10m")
- set(attributes["data_stream.dataset"], Concat([attributes["metricset.name"], "10m"], "."))
- set(attributes["processor.event"], "metric")
- duration: 60m
statements:
- set(resource.attributes["metricset.interval"], "60m")
- set(attributes["data_stream.dataset"], Concat([attributes["metricset.name"], "60m"], "."))
- set(attributes["processor.event"], "metric")
exporters:
debug: {}
# [Elasticsearch exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md)
Expand Down Expand Up @@ -906,17 +640,15 @@ collectors:
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug, signaltometrics, elasticsearch/otel]
exporters: [debug, elasticapm, elasticsearch/otel]
traces:
receivers: [otlp]
processors: [batch, elastictrace]
exporters: [debug, signaltometrics, elasticsearch/otel]
exporters: [debug, elasticapm, elasticsearch/otel]
metrics/aggregated-otel-metrics:
receivers:
- signaltometrics
- elasticapm
processors:
- batch/aggs
- lsminterval
exporters:
- debug
- elasticsearch/otel
Expand Down
Loading
Loading