Proposal for language additions to OTTL

[jsuereth]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

We were attempting to use OTTL to transform Google's structured logging format in GKE (see: https://cloud.google.com/logging/docs/structured-logging) to the current example data model in OpenTelemetry (see: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model-appendix.md#google-cloud-logging).

Effectively, we're trying to parse a JSON log body and extract components into OTLP equivalents:

{
  "severity":"ERROR",
  "message":"There was an error in the application.",
  "httpRequest":{
    "requestMethod":"GET"
  },
  "times":"2020-10-12T07:20:50.52Z",
  "logging.googleapis.com/insertId":"42",
  "logging.googleapis.com/labels":{
    "user_label_1":"value_1",
    "user_label_2":"value_2"
  },
  "logging.googleapis.com/operation":{
    "id":"get_data",
    "producer":"github.com/MyProject/MyApplication",
     "first":"true"
  },
  "logging.googleapis.com/sourceLocation":{
    "file":"get_data.py",
    "line":"142",
    "function":"getData"
  },
  "logging.googleapis.com/spanId":"000000000000004a",
  "logging.googleapis.com/trace":"projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824",
  "logging.googleapis.com/trace_sampled":false
}

In doing so we noticed a lot of friction in OTTL and duplicate expressions.

Here's a "simplified" (i.e. still missing some if/where statements, and new required built-in functions for span processing) version:

context: log
statements:
- set(body, ParseJSON(body["message"])) where (body != nil and body["message"] != nil)
- merge_maps(attributes, body["logging.googleapis.com/labels"], "upsert") where body["logging.googleapis.com/labels"] != nil
- delete_key(body, "logging.googleapis.com/labels") where (body != nil and body["logging.googleapis.com/labels"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/httpRequest"]) where (body != nil and body["logging.googleapis.com/httpRequest"] != nil)
- delete_key(body, "logging.googleapis.com/httpRequest") where (body != nil and body["logging.googleapis.com/httpRequest"] != nil)
- set(cache["value"], cache["__field_0"])
- set(attributes["gcp.http_request"], cache["value"]) where (cache != nil and cache["value"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/logName"]) where (body != nil and body["logging.googleapis.com/logName"] != nil)
- delete_key(body, "logging.googleapis.com/logName") where (body != nil and body["logging.googleapis.com/logName"] != nil)
- set(cache["value"], cache["__field_0"])
- set(attributes["gcp.log_name"], cache["value"]) where (cache != nil and cache["value"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/severity"]) where (body != nil and body["logging.googleapis.com/severity"] != nil)
- delete_key(body, "logging.googleapis.com/severity") where (body != nil and body["logging.googleapis.com/severity"] != nil)
- set(cache["value"], cache["__field_0"])
- set(severity_text, cache["value"]) where (cache != nil and cache["value"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/sourceLocation"]) where (body != nil and body["logging.googleapis.com/sourceLocation"] != nil)
- delete_key(body, "logging.googleapis.com/sourceLocation") where (body != nil and body["logging.googleapis.com/sourceLocation"] != nil)
- set(cache["value"], cache["__field_0"])
- set(attributes["gcp.source_location"], cache["value"]) where (cache != nil and cache["value"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/spanId"]) where (body != nil and body["logging.googleapis.com/spanId"] != nil)
- delete_key(body, "logging.googleapis.com/spanId") where (body != nil and body["logging.googleapis.com/spanId"] != nil)
- set(cache["value"], cache["__field_0"])
- set(span_id, cache["value"]) where (cache != nil and cache["value"] != nil)
- delete_key(cache, "__field_0") where (cache != nil and cache["__field_0"] != nil)
- set(cache["__field_0"], body["logging.googleapis.com/trace"]) where (body != nil and body["logging.googleapis.com/trace"] != nil)
- delete_key(body, "logging.googleapis.com/trace") where (body != nil and body["logging.googleapis.com/trace"] != nil)
- set(cache["value"], cache["__field_0"])
- set(trace_id, cache["value"]) where (cache != nil and cache["value"] != nil)

Describe the solution you'd like

We'd like to propose a new expression-focused syntax for OTTL that would allow the previous OTTL to look like this:

on log
when log.body is aStringMessage(parsedJson(json))
yield log with {
  attributes: attributes with json["logging.googleapis.com/labels"] with {
    "gcp.log_name": json["logging.googleapis.com/logName"]
  },
  spanID: StringToSpanID(json["logging.googleapis.com/spanId"]),
  traceID: StringToTraceID(json["logging.googleapis.com/trace"]),
  severity_text: json["logging.googleapis.com/severity"],
  body: json with {
    "logging.googleapis.com/labels": nil,
    "logging.googleapis.com/logName": nil,
    "logging.googleapis.com/severity": nil,
    "logging.googleapis.com/sourceLocation": nil,
    "logging.googleapis.com/spanId": nil,
    "logging.googleapis.com/trace": nil,
    "logging.googleapis.com/labels": nil,
  },
}

At a high level we propose the following:

• Supporting string-formatting literals, e.g. "I can reference {expression}s"
• Add a type system for better "prior to evaluation" error messages, including the ability to
  get error messages without running the collector. (e.g. Go, Rust, Typescript)
• Allow operations to operate against structural data, preferably with a JSON-like feel. (e.g. Jsonnet TypeScript, Dart)
  • Assign multiple values at the same time.
  • Have the visual structure mirrored in the code.
• Provide List comprehensions (e.g. Typescript, Python, Kotlin) that simplifies operating against
  lists of "KeyValueList" i.e. Attributes.
  • This should dramatically reduce need for built-in functions to be worthwhile.
  • This should replace need for: limit, truncate_all, replace_*, keep_keys, delete_keys.
• Pattern matching or "binding patterns", to simplify dealing with generic AnyValue attributes and log bodies, reducing the need to duplicate intent between where clause and statements.
• (optional) - Only reserve new keywords in "stateful" context. (This would require migrating to a stateful lexer, but could help avoid reducing the available identifiers).

Describe alternatives you've considered

We investigated leveraging CEL or LUA for this purpose.

Unfortunately there are a few shortcomings we think this proposal would alleviate:

• CEL lacks anonymous structural definitions. All key-value objects must be constructed as full types.
• CEL + LUA lack pattern-matching syntax. We believe pattern matching, with explicit "guard" vs. "execution" interpretation can dramatically simplify many OTTL expressions.
• CEL is inherently expression-based (returns a value) while OTTL is inherently imperative (mutates values)
• LUA is a full fledged programming language and could require more care to limit runtime complexity.

Additional context

I have a fully implemented prototype "trans-piler" which can take this new syntax and backport OTTL statements from it. This prototype includes grammar suggestions and rationale.

• Full Proposal
• Prototype Readme

I would like to consider whether OTTL should expand its expression prior to #28892.

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@jsuereth thanks for taking the time to dive into OTTL. There is a lot to comprehend here and it is going to take some time for me to ingest everything. I think this would be a very good topic for a Collector SIG meeting.

[TylerHelmuth]

@jsuereth since this isnt a PR yet I'm gonna supply some feedback from https://github.com/jsuereth/ottl-proposal/blob/main/PROPOSAL.md here

feedback on the complaints section

inconsistent and surprising syntax limitations (parentheses cannot be safely added around any expression, Boolean values can't be used as conditions without appending == true, time values can't be stored in cache fields, etc)

I believe all boolean or mathematical expressions can have () safely added as long as the result is still a boolean/mathematical expressiong. Do you have an example where this is not working.

Boolean values can now be used in conditions directly: #20911. Do you have an example where this is not working?

See #26108 for the cache and time values.

Isn't set(attributes["key"], nil) the same as delete_key(attributes, "key")

Currently set is no-op when the value is nil: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/func_set.go. I agree that isn't clear enough.

Inconsistency in identifier lookups. trace_id.string and trace_id["string"] don't refer to the same thing even though body.string and body["string"] do.

We've recently added a feature so that indexing fields that cannot be index throws an error on startup. body.string and body["string"] don't refer to the same thing: the former returns the body as a string and the latter requires the body to be a map and returns the value in the map at "string". I'll argue our path's have been consistent with the use of . to separate segments and what can and cannot be indexed.

feedback on the proposal section

• I'd absolutely love a type system. I've thought a little bit about how the context could tell the parse the type of a path and then how the parser could use that knowledge when creating the function, but I've never gotten far.
• I don't understand the pattern matching section yet. If the goal is to have a common where clause there is [processor/transform] Allow common where clause #27830, although I recognize that this doesn't solve the problem directly in OTTL
• At a brief glance I don't comprehend yet what these 2 expressions do, can you go into more detail?

  attributes: attributes with json["logging.googleapis.com/labels"] with {
   "gcp.log_name": json["logging.googleapis.com/logName"]
  },
  

  body: json with {
   "logging.googleapis.com/labels": nil,
   "logging.googleapis.com/logName": nil,
   "logging.googleapis.com/severity": nil,
   "logging.googleapis.com/sourceLocation": nil,
   "logging.googleapis.com/spanId": nil,
   "logging.googleapis.com/trace": nil,
   "logging.googleapis.com/labels": nil,
  },
  

[TylerHelmuth]

For the comparison to existing OTTL syntax, I'd like to submit my attempt at parsing a JSON log to specification. I probably misinterpreted some stuff, so please let me know if I've done something wrong:

Input:

{
  "severity":"ERROR",
  "message":"There was an error in the application.",
  "httpRequest":{
    "requestMethod":"GET"
  },
  "times":"2020-10-12T07:20:50.52Z",
  "logging.googleapis.com/insertId":"42",
  "logging.googleapis.com/labels":{
    "user_label_1":"value_1",
    "user_label_2":"value_2"
  },
  "logging.googleapis.com/operation":{
    "id":"get_data",
    "producer":"github.com/MyProject/MyApplication",
     "first":"true"
  },
  "logging.googleapis.com/sourceLocation":{
    "file":"get_data.py",
    "line":"142",
    "function":"getData"
  },
  "logging.googleapis.com/spanId":"000000000000004a",
  "logging.googleapis.com/trace":"projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824",
  "logging.googleapis.com/trace_sampled":false
}

transformprocessor config:

  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(cache, ParseJSON(body), "upsert") where IsMatch(body, "^\\{")

          - set(time, Time(cache["times"], "%Y-%m-%dT%H:%M:%S.%f%z")) where cache["times"] != nil

          - set(severity_number, SEVERITY_NUMBER_DEBUG) where cache["severity"] == "DEBUG"
          - set(severity_number, SEVERITY_NUMBER_INFO) where cache["severity"] == "INFO"
          - set(severity_number, SEVERITY_NUMBER_WARN) where cache["severity"] == "WARNING"
          - set(severity_number, SEVERITY_NUMBER_ERROR) where cache["severity"] == "ERROR"
          - set(severity_number, SEVERITY_NUMBER_FATAL) where cache["severity"] == "CRITICAL"

          - set(attributes["gcp.http_request"], cache["httpRequest"]) where cache["http_request"] != nil

          - set(span_id.string, cache["logging.googleapis.com/spanId"]) where cache["logging.googleapis.com/spanId"] != nil

          - replace_pattern(cache["logging.googleapis.com/trace"], "projects/.*/traces/([\\w\\d]+)", "$$1") where cache["logging.googleapis.com/trace"] != nil
          - set(trace_id.string, cache["logging.googleapis.com/trace"]) where cache["logging.googleapis.com/trace"] != nil

          - delete_key(cache, "times")
          - delete_key(cache, "severity")
          - delete_key(cache, "httpRequest")
          - delete_key(cache, "logging.googleapis.com/trace")
          - delete_key(cache, "logging.googleapis.com/spanId")
          - flatten(cache, prefix="gcp")
          - merge_maps(attributes, cache, "insert")

output:

Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2024-01-26 23:21:16.374373 +0000 UTC
Timestamp: 2020-10-12 07:20:50.52 +0000 UTC
SeverityText: 
SeverityNumber: Error(17)
Body: Str({"severity":"ERROR","message":"There was an error in the application.","httpRequest":{    "requestMethod":"GET"},"times":"2020-10-12T07:20:50.52Z","logging.googleapis.com/insertId":"42","logging.googleapis.com/labels":{    "user_label_1":"value_1",    "user_label_2":"value_2"},"logging.googleapis.com/operation":{    "id":"get_data",    "producer":"github.com/MyProject/MyApplication",    "first":"true"},"logging.googleapis.com/sourceLocation":{    "file":"get_data.py",    "line":"142",    "function":"getData"},"logging.googleapis.com/spanId":"000000000000004a","logging.googleapis.com/trace":"projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824","logging.googleapis.com/trace_sampled":false})
Attributes:
     -> logging.googleapis.com/spanId: Str(000000000000004a)
     -> times: Str(2020-10-12T07:20:50.52Z)
     -> logging.googleapis.com/sourceLocation: Map({"file":"get_data.py","function":"getData","line":"142"})
     -> httpRequest: Map({"requestMethod":"GET"})
     -> logging.googleapis.com/insertId: Str(42)
     -> logging.googleapis.com/labels: Map({"user_label_1":"value_1","user_label_2":"value_2"})
     -> message: Str(There was an error in the application.)
     -> logging.googleapis.com/operation: Map({"first":"true","id":"get_data","producer":"github.com/MyProject/MyApplication"})
     -> logging.googleapis.com/trace_sampled: Bool(false)
     -> log.file.name: Str(gcp.json)
     -> severity: Str(ERROR)
     -> logging.googleapis.com/trace: Str(projects/my-projectid/traces/06796866738c859f2f19b7cfb3214824)
     -> gcp.logging.googleapis.com/operation.id: Str(get_data)
     -> gcp.logging.googleapis.com/operation.producer: Str(github.com/MyProject/MyApplication)
     -> gcp.logging.googleapis.com/operation.first: Str(true)
     -> gcp.message: Str(There was an error in the application.)
     -> gcp.logging.googleapis.com/insertId: Str(42)
     -> gcp.logging.googleapis.com/labels.user_label_1: Str(value_1)
     -> gcp.logging.googleapis.com/labels.user_label_2: Str(value_2)
     -> gcp.logging.googleapis.com/trace_sampled: Bool(false)
     -> gcp.logging.googleapis.com/sourceLocation.file: Str(get_data.py)
     -> gcp.logging.googleapis.com/sourceLocation.line: Str(142)
     -> gcp.logging.googleapis.com/sourceLocation.function: Str(getData)
Trace ID: 06796866738c859f2f19b7cfb3214824
Span ID: 000000000000004a
Flags: 0
        {"kind": "exporter", "data_type": "logs", "name": "debug"}

I post this only for a more fair syntax comparison because I do not believe the one in the proposal is optimized.

[TylerHelmuth]

I'll also admit that my knee-jerk reaction is to be defensive haha I really appreciate the time you've spent on this proposal and your desire to improve OTTl. I promise to keep an open mind!

[jsuereth]

First, want to thank you for providing more optimal code for the use case! What I posted was our naive attempt with our current understanding of OTTL. It seems good that things can be simplified and I think some of our points were perhaps bugs that are now fixed as opposed to assumptions on syntax.

That said, I still think expanding available "expression" syntax and providing features like pattern matching or structural literals would be ideal to the types of programs you expect in OTTL.

I'm going to answer your questions over the next few days as I can and in separate comments, to hopefully have a good conversation and keep different points and features in separate conversation.

My proposal is here to help. While I stand behind all of it, we can try different pieces initially. I'm mostly looking to make sure when OTTL is stable it is scalable.

This isn't meant to be an attack on the current state, more a vision of where we can grow, and grow (hopefully) without breaking existing users.

[jsuereth]

At a brief glance I don't comprehend yet what these 2 expressions do, can you go into more detail?

attributes: attributes with json["logging.googleapis.com/labels"] with {
 "gcp.log_name": json["logging.googleapis.com/logName"]
},
body: json with {
 "logging.googleapis.com/labels": nil,
 "logging.googleapis.com/logName": nil,
 "logging.googleapis.com/severity": nil,
 "logging.googleapis.com/sourceLocation": nil,
 "logging.googleapis.com/spanId": nil,
 "logging.googleapis.com/trace": nil,
 "logging.googleapis.com/labels": nil,
},

This is my idea of a merge operation. The second is meant to be a mechanism of deleting multiple values from the JSON body. x with y is a merge maps method with insert. It takes the right and overrides the left with the value. The nice part of having this be an expression that returns a value is that you can chain it as I do in the first example.

The first examples does the following:

• inserts the nested key-value pairs from the JSON object found at key "googleapis.com/labels". These are expected to be string-string, but in my prototype parsejson returns the otlp AnyVal so you can basically assign that to attributes.
• inserts one specific attribute called "gcp log_name" pulled from a different area of the JSON.

The ability to chain the merges keeps the syntax brief but also visually you can see the key-value pairs being set.

[jsuereth]

I don't understand the pattern matching section yet. If the goal is to have a common where clause there is [processor/transform] Allow common where clause #27830, although I recognize that this doesn't solve the problem directly in OTTL

Pattern matching is designed to do two things:

• check some condition
• provide named terms you can use that are guaranteed to be safe.

The goal in this addition is not just sharing the conditional but also gaining a term (name? Reference?) You can use that has some desired type or feature.

Example from scala:

case class Data(x: Int, y: String)

val x: Any = ...
x match {
  case Data(_, name) => ...here I can use "name" as a string...
}

Example from rust:

enum Option<T> {
  None,
  Some(T)
}

match opt_value {
  None => handle_none(),
  Some(value) => handle_value(value),
}

The idea here is that you can chain these patterns too to dramatically simplify both checking a condition and adding a new term.

For OTTL I imagine we'd use this to verify log body AnyVal have specific fields of specific types and give names to those fields we can use in the program.

On log
When body like { some_field: String(name_ill_use) }
set(attributes.new_label, name_ill_use)

Here the user selected name_ill_use. The interpreter will verify some_field exists on body and that it contains a string value. That can be done in the guard/Boolean where clause.

Hope that helps! Let me know if you need more details or examples from other languages.

[TylerHelmuth]

My proposal is here to help. While I stand behind all of it, we can try different pieces initially. I'm mostly looking to make sure when OTTL is stable it is scalable. This isn't meant to be an attack on the current state, more a vision of where we can grow, and grow (hopefully) without breaking existing users.

Absolutely, I highlighting my own shortcomings haha we share the goal of making OTTL scalable and stable. Thanks again for thinking about this!

[TylerHelmuth]

@jsuereth in the examples where is the variable json coming from?

[TylerHelmuth]

@jsuereth I'd be really interested in seeing the examples from https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor#examples rewritten with this new syntax. I am curious how non-log/non-json statements look like and how they'll look in a yaml config.