Fleet : Kafka output : Dynamic Topic : Kibana wraps the custom topic name with a leading '%{[' and trailing ']}' pattern

[ChidamChidam]

Kibana version: 8.17.0 and 8.16.1

Elasticsearch version: 8.17.0 and 8.16.1

Server OS version: macOS Sequoia Version 15.2

Browser version: 131.0.6778.205

Browser OS version: macOS Sequoia Version 15.2

Original install method (e.g. download page, yum, from source, etc.):
docker installation of elasticsearch, kibana, fleet-agent, elastic-agent, kafka, zookeeper

Describe the bug:

Kibana introduces a leading '%{[' and trailing ']}' pattern to the custom topic name, which results in a "error":{"message":"failed to reload output: format string variables can not be nested in kafka.topic"} in the elastic-agent

Steps to reproduce:

In the Fleet > Settings > Add Output, choose Kafka and try the following options

1. Choose "Dynamic Topic" and key-in a custom topic name in the "Text Field" option say for example: %{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}, as in the image below

   

   In the Kibana logs you will see that new fleet output got added but the topic name is appended with these pattern "topics":[{"topic":"%{[%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}]}"}]

   Kibana Log

   [2024-12-20T07:07:59.608+00:00][DEBUG][elasticsearch.query.data] 201 - 210.0B
   PUT /.kibana_ingest_8.16.1/_create/ingest-outputs%3A57bb95ca-00e5-446e-a8b2-4f3df1b6a2fd?refresh=wait_for&require_alias=true
   {"ingest-outputs":{"name":"kafka","type":"kafka","hosts":["kafka:9093"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","proxy_id":null,"client_id":"Elastic","version":"2.6.0","compression":"none","auth_type":"none","connection_type":"plaintext","partition":"random","random":{"group_events":1},"topics":[{"topic":"%{[%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}]}"}],"headers":[{"key":"","value":""}],"timeout":30,"broker_timeout":30,"required_acks":1},"type":"ingest-outputs","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.6.0","updated_at":"2024-12-20T07:07:58.596Z","created_at":"2024-12-20T07:07:58.596Z"}
   [2024-12-20T07:07:59.609+00:00][DEBUG][plugins.fleet] Created new output

   When you assign this kafka output as the default output for Elastic Agent, it would fail with the following error

   Elastic Agent : Error Log : failed to reload output: format string variables can not be nested in kafka.topic

   {"log.level":"error","@timestamp":"2024-12-20T07:08:39.963Z","message":"could not start output","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-57bb95ca-00e5-446e-a8b2-4f3df1b6a2fd","type":"log"},"log":{"source":"log-57bb95ca-00e5-446e-a8b2-4f3df1b6a2fd"},"log.origin":{"file.line":662,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).reload"},"service.name":"filebeat","error":{"message":"failed to reload output: format string variables can not be nested in kafka.topic"},"ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}

2. Choose "Dynamic Topic" and key-in a custom topic name in the "Text Field" option say for example: data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace, as in the image below
   Note: We intentionally removed the leading '%{[' and trailing ']}' while keying-in the custom-topic

   

   Kibana again wraps the custom-topic with the leading and trailing pattern, but then this time since we intentionally removed it from the topic name, when kibana wraps it becomes a proper topic name.

   Kibana Log

   PUT /.kibana_ingest_8.16.1/_create/ingest-outputs%3Aac3be3a0-17b9-4907-b0d9-77163ce0befa?refresh=wait_for&require_alias=true
   {"ingest-outputs":{"name":"kafka","type":"kafka","hosts":["kafka:9093"],"is_default":false,"is_default_monitoring":false,"config_yaml":"","proxy_id":null,"client_id":"Elastic","version":"2.6.0","compression":"none","auth_type":"none","connection_type":"plaintext","partition":"random","random":{"group_events":1},"topics":[{"topic":"%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"}],"headers":[{"key":"","value":""}],"timeout":30,"broker_timeout":30,"required_acks":1},"type":"ingest-outputs","references":[],"managed":false,"coreMigrationVersion":"8.8.0","typeMigrationVersion":"10.6.0","updated_at":"2024-12-20T15:15:42.215Z","created_at":"2024-12-20T15:15:42.215Z"}
   [2024-12-20T15:15:42.350+00:00][DEBUG][plugins.fleet] Created new output ac3be3a0-17b9-4907-b0d9-77163ce0befa

   Elastic Agent Log
   No Errors in Elastic-Agent and we will see the custom topics being created in Kafka and the Data flows into those Custom Kafka topics.

Expected behavior:

Kibana should not wrap the custom-topic name with a leading '%{[' and trailing ']}' pattern

Errors in browser console (if relevant):

There are no errors in browser console, but the Elastic Agent fails to transport data to Kafka as discussed above

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

@criamico - Could you take a look at this one as you worked on this feature? I'm not sure if we're expecting a different format here that we should document or validate.

[cmacknz]

As far as I can tell, that the multi-format string approach here doesn't work is a limitation of the Fleet UI where the validation added to support dynamic topics doesn't account for there being more than one set of %{ } format string braces.

The agent implementation appears to support multiple format strings in the same text.

[criamico]

As far as I can tell, that the multi-format string approach here doesn't work is a limitation of the Fleet UI where the validation #192720 doesn't account for there being more than one set of %{ } format string braces.

Looking at the original ticket and the docs, neither of them mentioned these multiple format strings. We'll need to adjust the fleet validation to allow them.

[cmacknz]

Looking at the https://github.com/elastic/ingest-dev/issues/3407#issuecomment-2262970947 and the docs, neither of them mentioned these multiple format strings.

I also learned we had this feature from this bug report :)

[rz1027]

A question here, can I set output for monitoring as Kafka (and use dynamic topic) ??

[nimarezainia]

A question here, can I set output for monitoring as Kafka (and use dynamic topic) ??

@rz1027 You can do this correct.

[Supplementing]

@criamico @kpollich (& maybe @cmacknz) Working on this fix now, and looks like there may be a few approaches here so wanted to get more context and input from the team before I decide on the implementation.

Option 1. We can add validation to the field to stop the leading and trailing characters (which cause the issue) from being entered by the user. The field would go into an error state, show a message, and stop submission. Sounds like this isn't the best though as we want to support the multiple format strings and then it looks a bit strange with the leading/trailing brackets missing while the rest of the string has them throughout.

Option 2. We can remove the hardcoded wrapping of the entered string as proposed above in the issue itself (which adds %{{[ and ]} to the start and end of any custom dynamic topic that is added. However, doing this would also require an update to the text field validation making the user required to manually enter those characters every time. Theyre already entering them throughout the string so this could work. It does have the drawback of needing more validation in general though to ensure that all the brackets are closed correctly, otherwise it could lead to more errors I think.

Option 3. We do a quick check of what was entered before assigning the topic value, and if it starts and ends with the characters, do nothing, otherwise, add them. This one also needs the same check as option 2 to make sure the brackets match up in pairs across the entire entered string. **I think I prefer this option over the others as it seems most flexible. **

[kpollich]

Option 3. We do a quick check of what was entered before assigning the topic value, and if it starts and ends with the characters, do nothing, otherwise, add them. This one also needs the same check as option 2 to make sure the brackets match up in pairs across the entire entered string. **I think I prefer this option over the others as it seems most flexible. **

I'm on board with this option as well. It seems relatively straightforward for us to make the input "intelligent" in this way.

[kpollich]

Hmm thinking about it again I wonder if adding the wrapping characters universally is going to cause other issues. If I want to use a dynamic topic of my-topic-%{[data_stream.namespace]} would I wind up with %{[my-topic-%{[data_stream.namespace]}]} and therefore have the same syntax error?

I don't think this "mix and match" style identifier would be covered here. Maybe we should simply be less opinionated about what users put into this input and not try to wrap things for them. We can provide hint text for the expected syntax.