Inject Beat gRPC stream identifiers without relying on agent specification file transformations

[cmacknz]

To quickly allow metricbeat and filebeat to integrate with the shipper the stream identifiers (data stream and input ID) needed to allow the shipper to map events to processors are being injected via the inject_stream_processor specification file transformation. See the implementation in elastic/elastic-agent#527.

In the V2 agent specification file format the agent will no longer be able automatically transform sections of the agent policy into a beat configuration file. All inputs must know how to configure themselves to run under agent automatically.

Define and implement a way of providing the needed stream identifiers to agent inputs without relying on specification file transformations as required by the V2 specification file format. This includes all beats (not just filebeat and metricbeat) and non-beat agent inputs like endpoint security.

Ideally the beats can consume the agent policy sections containing the relevant information directly without transformation, and instantiate the necessary processors automatically. This is similar to the approach heartbeat currently uses, see the heartbeat specification file and heartbeat code adding the processors.

Relates To:

• [Meta] Remove Elastic Agent V1 control protocol elastic-agent-client#32
• [Meta][Feature] The agent policy's input processor configurations must be provided to the shipper. elastic-agent#220
  • Provide the relevant subset of the agent policy to the shipper as its configuration elastic-agent#617

This task is considered complete when:

1. A test exists proving that the shipper can map processors to events sent from filebeat and metricbeat without using any configuration file transformations.
2. The implementation used for metricbeat and filebeat is available for reuse in libbeat or elastic-agent-libs, such that any existing beat can easily be updated to add the new stream identifiers to produced events when communicating with the shipper.