-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add blog post about the OTTL context inference feature #6290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
cartermp
merged 14 commits into
open-telemetry:main
from
edmocosta:add-ottl-contexts-just-got-easier-blogpost
Feb 19, 2025
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
eaceb27
Add OTTL context inference blogpost
edmocosta 03524b8
Update refcache.json
edmocosta 3236c44
Apply suggestions from code review
edmocosta 77c60fb
Merge branch 'main' into add-ottl-contexts-just-got-easier-blogpost
edmocosta 95b5b2e
Fix format
edmocosta 4e75d4a
Changed to emphasizing the flat configuration style first
edmocosta 17fe023
Apply review suggestion
edmocosta 241b8bd
Apply suggestions from code review
edmocosta 2e80eb8
Add an extra section comparing the configuration styles
edmocosta 9804ee6
Update refcache.json
edmocosta 8ace825
Apply suggestions from code review
edmocosta fc55ccc
Update date to 2025-02-20
edmocosta ea80b16
Update config names and links
edmocosta fd919d7
Remove draft flag
edmocosta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,155 @@ | ||
--- | ||
title: OTTL contexts just got easier with context inference | ||
linkTitle: OTTL contexts just got easier | ||
date: 2025-02-20 | ||
author: '[Edmo Vamerlatti Costa](https://github.com/edmocosta) (Elastic)' | ||
issue: 6289 | ||
sig: Collector SIG | ||
cSpell:ignore: OTTL Vamerlatti | ||
--- | ||
|
||
Selecting the right OTTL context for running statements can be challenging, even | ||
for experienced users. Choosing the correct option impacts both accuracy and | ||
efficiency, as using higher-level OTTL contexts can avoid unnecessary iterations | ||
through nested lower-level contexts. | ||
|
||
To simplify this process, the OpenTelemetry community is excited to announce | ||
OTTL | ||
[context inference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#context-inference) | ||
support for the | ||
[transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor). | ||
This feature removes the need for users to understand the underlying context | ||
concept of OTTL, allowing them to focus solely on their data. It also improves | ||
statement processing efficiency by automatically selecting the most appropriate | ||
OTTL context. This optimization ensures that data transformations are both | ||
accurate and performant. | ||
|
||
## How does it work? | ||
|
||
Starting with version `0.120.0`, the transform processor supports two new | ||
context-inferred configuration styles. The first one offers a simpler and | ||
flatter approach, while the second closely resembles the existing configuration | ||
format. | ||
|
||
### Basic configuration | ||
|
||
The | ||
[basic configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#basic-config) | ||
style simplifies configuration by allowing users to list all statements | ||
together, without worrying about OTTL contexts or extra configuration | ||
structures. This style support statements from multiple OTTL contexts and does | ||
not require grouping them separately. | ||
|
||
To illustrate this, compare the following configuration: | ||
|
||
```yaml | ||
metric_statements: | ||
- context: resource | ||
statements: | ||
- keep_keys(attributes, ["host.name"]) | ||
- context: metric | ||
statements: | ||
- set(description, "Sum") where type == "Sum" | ||
- convert_sum_to_gauge() where name == "system.processes.count" | ||
- context: datapoint | ||
statements: | ||
- limit(attributes, 100, ["host.name"]) | ||
``` | ||
|
||
With the new basic configuration style, the same logic is expressed more | ||
concisely by simply providing a list of statements: | ||
|
||
```yaml | ||
metric_statements: | ||
- keep_keys(resource.attributes, ["host.name"]) | ||
- set(metric.description, "Sum") where metric.type == "Sum" | ||
- convert_sum_to_gauge() where metric.name == "system.processes.count" | ||
- limit(datapoint.attributes, 100, ["host.name"]) | ||
``` | ||
|
||
This streamlined approach enhances readability and makes configuration more | ||
intuitive. To use this configuration style, all paths in the statements must be | ||
prefixed with their respective OTTL contexts. These prefixes are required for | ||
all context-inferred configurations and serve as hints for selecting the best | ||
match. It also makes statements unambiguous and portable between components | ||
using OTTL. | ||
|
||
### Advanced configuration | ||
|
||
The context-inferred | ||
[advanced configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md#advanced-config) | ||
style closely resembles the existing format and allows users to leverage the | ||
benefits of context inference while providing granular control over statement | ||
configurations, such as `error_mode` and `conditions`. For example, consider the | ||
following configuration: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- context: datapoint | ||
conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") where attributes["my.attr"] == "some" | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
The above can now be written as: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") where datapoint.attributes["my.attr"] == "some" | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
In this example, the `context` value is omitted and is automatically inferred to | ||
`datapoint`, as it is the only OTTL context present in the statements that | ||
supports parsing both `datapoint` and `metric` data. | ||
|
||
If we update the above configuration removing the `datapoint` usage: | ||
|
||
<!-- prettier-ignore-start --> | ||
```yaml | ||
metric_statements: | ||
- conditions: | ||
- resource.attributes["service.name"] == "my.service" | ||
statements: | ||
- set(metric.description, "counter") | ||
``` | ||
<!-- prettier-ignore-end --> | ||
|
||
The context inferrer would select the `metric` OTTL context instead, since no | ||
data points are accessed. Although it would be possible to run the statements | ||
using the `datapoint` OTTL context, `metric` is the most efficient option. | ||
|
||
### Which configuration style should I choose? | ||
|
||
The [basic configuration](#basic-configuration) style is best suited for | ||
scenarios where simplicity and ease of use are paramount. It is ideal for simple | ||
use cases where your configuration needs are straightforward and do not require | ||
the use of additional configurations keys, allowing you to quickly set up your | ||
statements with minimal effort and without needing to understand the underlying | ||
concept of OTTL contexts. | ||
|
||
The [advanced configuration](#advanced-configuration) style is more detailed and | ||
allows the use of additional configuration keys such as `error_mode` and | ||
`conditions`. It supports statements from multiple OTTL contexts. However, | ||
unlike the basic configuration style, it may require splitting them into | ||
separate configuration groups (see | ||
[advanced config](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor#advanced-config)). | ||
In terms of performance, the advanced configuration is slightly faster than the | ||
basic configuration, making it a better choice for complex scenarios or | ||
configurations with a high number of statements. | ||
|
||
## Try it out | ||
|
||
As we wrap up, we encourage users to explore this new functionality and take | ||
advantage of its benefits in their telemetry pipelines! | ||
|
||
If you have any questions or suggestions, we’d love to hear from you! Join the | ||
conversation in the `#otel-collector` channel on the | ||
[CNCF Slack workspace](https://slack.cncf.io/). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.