[Elastic Agent] Add ability to debug a composable provider

[blakerouse]

Overview

To help support dynamic inputs elastic/beats#19225 Elastic Agent needs to add the ability to debug the providers using for variable substitution. This issue is to track the debugging effort, for information about variable substitution review elastic/beats#20781

Debugging

This obviously adds a lot of confusion to what the resulting configuration that Elastic Agent will be running with. To ensure that the feature is deployed correctly and that providers are working as expected debugging needs to be a top priority in the implementation.

Debugging the running daemon

With the new ability to communicate with the running daemon the inspect command should be changed to talk to the running daemon and return the current configuration that is being used in memory. This will ensure that with running providers like Docker and Kubernetes it is easy to inspect what the resulting configuration is.

The current inspect and output commands can be combined and moved under the debug subcommand. (Note: This is not connecting to the currently running Elastic Agent)

$ ./elastic-agent debug config

Possible to watch the configuration as changes come in with --watch.

$ ./elastic-agent debug config --watch

Debugging a single provider

A new debug command should be implemented that runs a single provider and output what it's currently providing back to the Elastic Agent. (Note: This is not connecting to the currently running Elastic Agent)

Example outputting docker provider inventory key/value mappings:

$ ./elastic-agent debug provider docker 
{"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}},}
{"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}},}
{"id": "2", "mapping": nil}

Example rendering configurations with changes:

$ ./elastic-agent debug provider docker -c testing-conf.yml
# {"id": "1",  "mapping": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}, "processors": {"add_fields": {"container.name": "my-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
# {"id": "2", "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}}, "processors": {"add_fields": {"container.name": "other-container"}}}
inputs:
  - type: logfile
    path: /var/log/containers/1.log
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: /var/log/containers/2.log
    processors:
      - add_fields:
          container.name: other-container
# {"id": "2", "mapping": nil}
inputs:
  - type: logfile
    path: /var/log/containers/1.log

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

++ on the proposal. I would skip the --watch part at first probably. I really like that the state of a provider can be read out and also the current running config.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@michel-laterman was it part of your diagnosis job? If yes, can we close this?`

[michel-laterman]

The scope of the outlined issue is more then what is currently included with diagnostics, let's keep this open as I think we should add this to the command (eventually)

[michel-laterman]

I want to add a little more details here as it relates to diagnostics.

Currently diagnostics will use the exact same output of inspect (and inspect output) when gathering config from the agent and running processes.
diagnostics also gathers its own copy of the local (elastic-agent.yml/fleet.yml) config files (I think with variable substitution?).

The diagnostics command should provide the process config/policies/etc from the running agent.

However, I don't think we should name the command that does not contact from to the running agent debug as we tend to use diagnostics and debug interchangeably; maybe add it as a diagnostic subcommand, or add a flag to inspect?