Tenzir Node v5.23.0

This release introduces centralized node-level TLS configuration, allowing you to configure TLS settings once in tenzir.yaml instead of passing options to each operator individually. It also adds support for event-timestamp-based compaction rules and a count field in the deduplicate operator.

🚀 Features

Count dropped events in deduplicate operator

The deduplicate operator now supports a count_field option that adds a field to each output event showing how many events were dropped for that key.

Example

from {x: 1, seq: 1}, {x: 1, seq: 2}, {x: 1, seq: 3}, {x: 1, seq: 4}
deduplicate x, distance=2, count_field=drop_count

{x: 1, seq: 1, drop_count: 0}
{x: 1, seq: 4, drop_count: 2}

Events that are the first occurrence of a key or that trigger output after expiration have a count of 0.

By @raxyte in #5622.

Node-level TLS configuration for operators

All operators and connectors that use TLS now support centralized node-level configuration. Instead of passing TLS options to each operator individually, you can configure them once in tenzir.yaml under tenzir.tls.

Arguments passed directly to the operator itself via an argument take precedence over the configuration entry.

The following options are available:

• enable: Enable TLS on all operators that support it
• skip-peer-verification: Disable certificate verification
• cacert: Path to a CA certificate bundle for server verification
• certfile: Path to a client certificate file
• keyfile: Path to a client private key file
• tls-min-version: Minimum TLS protocol version ("1.0", "1.1", "1.2", or "1.3")
• tls-ciphers: OpenSSL cipher list string

The later two options have also been added as operator arguments.

For server-mode operators (load_http server=true, load_tcp), mutual TLS (mTLS) authentication is now supported:

• tls-client-ca: Path to a CA certificate for validating client certificates
• tls-require-client-cert: Require clients to present valid certificates

These two options are also available as operator arguments.

Example configuration enforcing TLS 1.2+ with specific ciphers:

tenzir:
  tls:
    tls-min-version: "1.2"
    tls-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"
    cacert: "/etc/ssl/certs/ca-certificates.crt"

By @IyeOnline in #5559.

Platform TLS configuration

The Tenzir Node now lets you configure the minimum TLS version and TLS ciphers accepted for the connection to the Tenzir Platform:

plugins:
  platform:
    tls-min-version: "1.2"
    tls-ciphers: "HIGH:!aNULL:!MD5"

By @lava in #5559.

Use event timestamps for compaction rules

Compaction rules can now use event timestamps instead of import time when selecting data by age. Configure this using the new optional field key in the compaction configuration.

Previously, compaction always used the import time to determine which partitions to compact. Now you can specify any timestamp field from your events:

tenzir:
  compaction:
    time:
      rules:
        - name: compact-old-logs
          after: 7d
          field: timestamp  # Use event timestamp instead of import time
          pipeline: |
            summarize count=count(), src_ip

When field is not specified, compaction continues to use import time for backward compatibility.

By @jachris in #5629.

🐞 Bug fixes

Fixed default compaction rules for metrics and diagnostics

The default compaction rules for tenzir.metrics.* and tenzir.diagnostic events now correctly use the timestamp field instead of import time.

Previously, these built-in compaction rules relied on import time to determine which events to compact, which could lead to inconsistent results as the import time is not computed per-event. As a result, it was possible that metrics and diagnostics were not deleted even though they expired.

By @jachris in #5629.

Tenzir Node v5.22.2

This release fixes a performance regression when parsing lists with mixed-type elements, where batch processing was inadvertently broken. It also resolves an assertion failure that could crash Tenzir when encountering events with duplicate keys.

🐞 Bug fixes

Fixed assertion failure when encountering duplicate keys

We fixed an assertion failure and subsequent crash that could occur when parsing events that contain duplicate keys.

By @IyeOnline in #5612.

Improved Type Conflict Handling

We resolved an issue that would appear when reading in lists (e.g. JSON []) where the elements had different types. Tenzir's type system at this time only supports storing a single type in a list. Our parsers resolve this issue by first attempting conversions (e.g. to a common numeric type) and turning all values into strings as a last resort. Previously this would however also break Tenzir's batch processing leading to significant performance loss. This has now been fixed.

By @IyeOnline in #5612.

Publishing Bugfix

This release fixes a bug where the publish operator could drop events.

For a detailed list of all changes, view the changelog.

UDO Arguments

This release introduces support for arguments in user-defined operators, letting operators declare positional and named parameters with optional default values and use them just like built-in operators. It also enhances parser behavior for duplicate keys and includes several important stability, parsing, and retention improvements to make pipelines more flexible and reliable.

For a detailed list of all changes, view the changelog.

Operator Fixups

This is a bugfix release that fixes timestamp handling in the python operator and the backpressure handling in publish and subscribe.

For a detailed list of all changes, view the changelog.

Backport Release

This release backports the changes made in v5.21.1 to v5.20.1.

For a detailed list of all changes, view the changelog.

New Hash Functions

This release features new and improved hash functions as well as a couple of bugfixes.

For a detailed list of all changes, view the changelog.

Better Backpressure

This release improves the stability of pipelines by applying backpressure more effectively, ensuring upstream components slow down before overwhelming subscribers. It also enhances daily operations by improving parquet reliability, adding Base58 support, and extending the built-in OCSF operators and functions to support the newly released OCSF 1.7.0.

For a detailed list of all changes, view the changelog.

More Memory Metrics

This patch release comes with a few new experimental memory metrics. Furthermore, it resolves an issue where the memory usage would grow without bounds on some systems.

For a detailed list of all changes, view the changelog.

Better Memory Allocation

We continue our quest for better memory usage by switching out the memory allocator to the battle-tested mimalloc, as well as adding metrics collecting for memory usage.

For a detailed list of all changes, view the changelog.