[8.19] (backport #10087) Fall back to process runtime if otel runtime is unsupported

[mergify]

What does this PR do?

Makes inputs use the process runtime if they're configured to use the otel runtime, and the latter cannot support them. Currently there are two possible reasons for this:

• The output is not suppored at all - logstash is an example.
• The output configuration uses unsupported config options - allow_older_versions: false for the elasticsearch output for example.

A log line is also emitted if this happens.

I've also moved the component monitoring code into its own package to avoid dependency cycles.

Why is it important?

We want to enable the otel runtime progressively without ever breaking a user's working setup. This means that a fallback is necessary, even if it may involve going against the explicit configuration. Later, we may add a flag which causes this to be an error instead.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Build the agent package and run it locally with either an elasticsearch output using an unsupported option (this PR uses allow_older_versions: false), or a kafka or logstash output. Then look at the status and logs. You should see a log line warning that your input was switched to the process runtime.

Related issues

• Closes [beats receivers] Only run components with supported outputs as Beats receivers #9696
• Closes [beats receivers] Conditionally enable Beats receivers only if configuration translation succeeds #9098

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #10087 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 2f0ba69 has failed:

On branch mergify/bp/8.19/pr-10087
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 2f0ba69f2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   internal/pkg/agent/application/coordinator/coordinator_unit_test.go
	renamed:    internal/pkg/agent/application/monitoring/testdata/monitoring_config_full.yaml -> internal/pkg/agent/application/monitoring/component/testdata/monitoring_config_full.yaml
	renamed:    internal/pkg/agent/application/monitoring/v1_monitor.go -> internal/pkg/agent/application/monitoring/component/v1_monitor.go
	renamed:    internal/pkg/agent/application/monitoring/v1_monitor_test.go -> internal/pkg/agent/application/monitoring/component/v1_monitor_test.go
	modified:   internal/pkg/agent/application/monitoring/process.go
	modified:   internal/pkg/agent/application/monitoring/server.go
	modified:   internal/pkg/agent/cmd/inspect.go
	modified:   internal/pkg/otel/manager/diagnostics.go
	modified:   internal/pkg/otel/manager/diagnostics_test.go
	modified:   testing/integration/ess/beat_receivers_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/agent/application/application.go
	both modified:   internal/pkg/agent/application/coordinator/coordinator.go
	both modified:   internal/pkg/otel/translate/otelconfig.go
	both modified:   internal/pkg/otel/translate/otelconfig_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c888491

Failed CI Steps

• arm:sudo: upgrade

History

• 💔 Build #27592 failed e70295a

cc @swiatekm