Add Dynamic index

[flenoir]

Component(s)

exporter/opensearch

Is your feature request related to a problem? Please describe.

I was using elasticsearch before but now i moved to opensearch. I wanted to use same dynamic index feature in order to create index per day.
Unfortunately not possible with opensearch exporter. Will it be added in the futur ?

Describe the solution you'd like

adding dynamic index like in elasticsearch exporter

Describe alternatives you've considered

Is there another way to add this daily rotation in order to have index like "my_index-20608-2024" for example

Thanks

Additional context

none

[github-actions]

Pinging code owners:

• exporter/opensearch: @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

you meaning logstash_format feature in elasticsearchexporter?

[codingbott]

you meaning logstash_format feature in elasticsearchexporter?

He is also talking about defining the name of datastream created by the exporter.
He expects that the opensearch exporter has the same feature as ElasticSearch exporter.
In that one you can build a dynamic index name according to attributes from the log record.

I'm using that feature to specify the index to write the log to, from the oidc token.

Take a look here, please: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md
Search for "logs_dynamic_index".

here is the extract of that:

• logs_dynamic_index (optional): uses resource, scope, or log record attributes to dynamically construct index name.
  • enabled(default=false): Enable/Disable dynamic index for log records. If data_stream.dataset or data_stream.namespace exist in attributes (precedence: log record attribute > scope attribute > resource attribute), they will be used to dynamically construct index name in the form logs-${data_stream.dataset}-${data_stream.namespace}. Otherwise, if
    elasticsearch.index.prefix or elasticsearch.index.suffix exist in attributes (precedence: resource attribute > scope attribute > log record attribute), they will be used to dynamically construct index name in the form ${elasticsearch.index.prefix}${logs_index}${elasticsearch.index.suffix}. Otherwise, if scope name matches regex /receiver/(\w*receiver), data_stream.dataset will be capture group Create CODEOWNERS #1. Otherwise, the index name falls back to logs-generic-default, and logs_index config will be ignored. Except for prefix/suffix attribute presence, the resulting docs will contain the corresponding data_stream.* fields, see restrictions applied to Data Stream Fields.

[EzzioMoreira]

@flenoir did you resolve it? I will need to implement the creation dynamic index on Opensearch.

[njuyen]

+1 on this

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[ps48]

@flenoir We'll start working on this feature soon as we revive the OpenSearch Exporter. Can you please share some specifics on the feature requirements? This will help us understand how to provide the dynamic index feature along with the existing SS4O support.

[OldTyT]

@flenoir We'll start working on this feature soon as we revive the OpenSearch Exporter. Can you please share some specifics on the feature requirements? This will help us understand how to provide the dynamic index feature along with the existing SS4O support.

Hello!
Thank you so much for developing such a cool solution!
From what I’ve understood about this integration, the following features are missing:

• Logstash format. The ability to specify year/month/day in the index, e.g., myapp-2025.05.19 (ideally, with the option to customize the date format). This is needed for configuring rotation.
• Dynamic index selection. For example, changing the index name based on the service name sending the logs. Use case: restricting access to logs by separating access to index patterns.

[sw90lee]

Hello
I'd like to request assignment to the feature implementation that allows specifying a Logstash-style index format with date components.
I’m interested in contributing this functionality and would be happy to follow the existing contribution guidelines. Please let me know if this can be assigned to me.

Thanks!