[chore][RFC] - Configuration Merging revamped

[VihasMakwana]

Description

This RFC is a follow-up of #12097. The first PR introduced the feature gate to merge the components' lists and left out the options to configure the merging behaviour.

This RFC proposes an approach to extend the current behaviour by enabling merging of specified config parts and support different modes.

Link to tracking issue

Relates:

• Append components flag #8754
• Allow instantiating extensions in separate configs #8394
• [ConfMap] - Providing custom merge method for koanf's Map.Merge #10370

Thanks to @mx-psi @dmitryax and @evan-bradley for their feedback on the first PR!!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.38%. Comparing base (3bcdf80) to head (21caaf8).
⚠️ Report is 323 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13256      +/-   ##
==========================================
- Coverage   91.62%   91.38%   -0.25%     
==========================================
  Files         522      646     +124     
  Lines       29208    42606   +13398     
==========================================
+ Hits        26763    38937   +12174     
- Misses       1926     2844     +918     
- Partials      519      825     +306     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[douglascamata]

Good stuff, @VihasMakwana. I like this. I left comments but they are non-blockers to me.

[dmitryax]

We probably need more options. For example, if I want to add a receiver in a merging config while it may be already defined in the base config. Also, we need the default replace behavior. What about the following list:

• append: append entries to the list
• append_unique: append only unique entries to the list
• prepend: insert entries in front of the list
• prepend_unique: insert only unique entries in front of the list
  replace (default): overwrite the list
  keep: keep the original list

Also, I would call the options list_merge_mode and list_merge_paths to make it clearer.

[VihasMakwana]

@dmitryax I see. I'll update the RFC

[douglascamata]

I think you touch a good point there with the _unique modifiers, @dmitryax. Shouldn't both append and prepend always ensure lists containing only "basic types" (strings, ints, etc) do not have duplicated values? I don't know about any scenario in the Collector involving lists of basic types where duplicated values wouldn't be a problem.

[VihasMakwana]

For service components, duplicates will cause error.
I'm not sure about other places though.

[dmitryax]

For service components, duplicates will cause error.

Right. For example, I have a list of receivers in a basic config file which I don't control. Then, I want to add another receiver and override it in case it's already defined in the basic config. In that case --config=extra_receiver.yaml?merge_mode=append&merge_paths=service::**::receivers would cause a failure if the basic config already has my receiver. I need append_unique instead.

[mx-psi]

I think we can start with a small number of options, in any case we can discuss that in the implementation phase

[dmitryax]

I think using the query params is fine. However, we need to keep in mind that some confmap providers use the same pattern to pass extra information. With this approach, they won't be able to use these keys which is probably fine.

If anyone else have any other ideas, please share.

cc @open-telemetry/collector-approvers

[VihasMakwana]

yeah, we can extract the parameters here

opentelemetry-collector/confmap/resolver.go

Line 176 in 6aa2b81

for _, uri := range mr.uris {

and process them for our merge function.

Then we can remove our parameters from URI and let resolver do its job:

u, _ := url.Parse("otel.yaml?key=val&merge_mode=append")
q := u.Query()
q.Del("merge_mode")
u.RawQuery = q.Encode()
// uri will now be otel.yaml?key=val&key2=val2

[evan-bradley]

We discussed this in this RFC and came up with a few issues with query parameters that we'll want to consider before going that route.

@dmitryax I believe you, @mx-psi, and I had an offline conversation at KubeCon and tacitly agreed we should go the route of configuring these inside a config file (this option in the RFC). Do you still think we should go this route? I think we should probably at least explore it before deciding on an approach.

[mx-psi]

@evan-bradley @dmitryax approach 1 does not deal with params, does this resolve your concerns here?

[evan-bradley]

I wouldn't mind calling out the "Configuring Confmap Providers" RFC just to make it clear there are other options we can explore, but if approach 1 is our preference, I'm good with what's here.

[mx-psi]

Yes, that's fair

[mx-psi]

I haven't had time to look into this proposal in detail but one thing I would like to see explored is whether we can use YAML custom tags https://yaml.org/spec/1.2.2/#tags to specify this in the YAML itself per-array

[VihasMakwana]

@mx-psi Thanks for sharing your thoughts! I'll surely go the documentation and see if they can be integrated. I'm all in if we can get it work through just yaml.

[VihasMakwana]

@mx-psi I had a brief look at yaml tags and their support in golang. Here are my thoughts:

1. We rely on koanf for configuration management and it deals with map[string]any. It does not support the concept of yaml tags. To implement this feature using yaml tags, we need to extract tags somewhere around here

   opentelemetry-collector/confmap/provider.go

   Lines 154 to 159 in 18b3b57

   	// NewRetrievedFromYAML returns a new Retrieved instance that contains the deserialized data from the yaml bytes.
   	// * yamlBytes the yaml bytes that will be deserialized.
   	// * opts specifies options associated with this Retrieved value, such as CloseFunc.
   	func NewRetrievedFromYAML(yamlBytes []byte, opts ...RetrievedOption) (*Retrieved, error) {
   	var rawConf any
   	if err := yaml.Unmarshal(yamlBytes, &rawConf); err != nil {

• One thing to note: We use https://pkg.go.dev/sigs.k8s.io/yaml, which is a fork of gopkg.in/yaml.v3. I believe it should support yaml tags. I'll get back to you if I face any blockers.
• Pseudo code for extracting tags:

var node Node
err := yaml.Unmarshal(data, &node)

// recursively go through the node and its children to find the nodes with tags and store their path.

2. I still need to create a proof of concept to be sure it works. I'm planning to update the RFC with both approaches and decide on the best one after evaluating the pros and cons of each.

Here are some initial pros and cons that come to mind:

1. Pros of yaml tag approach:

   • Straightforward from end-user's perspective. No need to extra CLI flags and everything can be specified in configuration files.

2. Cons of yaml tag approach:

   • More preprocessing:
     • As koanf converts the given config into map[string]any, we need to do the preprocessing of tags before converting it to a koanf instance or else we'll lose the tags.
     • We need to go down the yaml tree to extract nodes with tags.

Let me know your thoughts!

[VihasMakwana]

@mx-psi I just saw your raised #13308. That makes things a lot easier for yaml tags. Thanks!

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[mx-psi]

I discussed this over Slack with Vihas but mentioning it here: it would be good to make sure that our officially released Helm charts and operator do not break these YAML tags (they shouldn't but I think it's a good check to do before deciding)

[VihasMakwana]

I've reached out on #otel-helm on CNCF slack for more eyes on this. I'll run some tests on this and keep you updated.

[VihasMakwana]

@mx-psi Unfortunately, YAML tags are not preserved in helm chart. I think this is due to the fact that our helm chart deals with yaml to dict conversion internally and tags are lost in this process.

The same applies to the operator. The operator marshals the config into a struct and internally, it is map[string]interface{}. Hence, the tags are lost in the conversion.

That being said, do users need this feature in helm/operator? Our helm chart internally deals with all the merging and appending to the lists, when necessary.

For operator, users deploy a single configuration like following:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [debug]
EOF

[VihasMakwana]

^^ @mx-psi

[mx-psi]

From 2025-09-10 Collector SIG meeting it seems like this should not be a concern but let's wait a bit more for the Operator SIG to answe

[VihasMakwana]

cc: @evan-bradley @dmitryax

if you guys are around, please take a look at this RFC. I've documented both the approaches.

Here's a PoC for yaml tags: #13551

[VihasMakwana]

@mx-psi if you're around, can you please take a look?

[mx-psi]

LGTM, let's wait to see what the Operator SIG says but I think the suggested approach works and would be my preferred alternative

[mx-psi]

@dmitryax @evan-bradley could you take another look so we can know if there are any outstanding discussions to be had before going forward with this?

[evan-bradley]

Overall looks good to me, thanks @VihasMakwana.

[paulojmdias]

Hi @VihasMakwana, great work, fantastic stuff! 💪

Do you think the points raised in this issue might also be worth evaluating for inclusion in this RFC (or perhaps in a follow-up)?

The use case is a bit different, but there seems to be real value for users in merging configurations dynamically from multiple files within a directory.

[VihasMakwana]

@paulojmdias I like that enhancement! It can definitely be a follow-up in my opinion.

[paulojmdias]

@paulojmdias I like that enhancement! It can definitely be a follow-up, in my opinion.

Awesome, let's see what other people think about it. I'm also open to help with it 🙌

[mx-psi]

This was discussed in the last Operator SIG meeting with no major concerns. I think we can merge this on or after Wednesday unless we hear any objections

[swiatekm]

Speaking on behalf of the operator SIG, we're unlikely to use this feature, so no objections from our side.

[douglascamata]

Lovely so see this merged, @VihasMakwana. Congrats and thank you!

Do you have an implementation plan in mind? Are there are issues or PRs that I can follow to know when work starts to move forward so that I help out and/or know when it's ready to be used?

[VihasMakwana]

@douglascamata Hello!

Do you have an implementation plan in mind? Are there are issues or PRs that I can follow to know when work starts to move forward so that I help out and/or know when it's ready to be used?

You can keep an eye on #13551. It's still in draft and I need to clean things up. I'll find some time this week/next one and mark it for review.