[receiver/receiver_creator] Add support for logs' hints

Signed-off-by: ChrsMark <[email protected]>

Author: ChrsMark

.chloggen/f_hints_logs.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: receivercreator
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add support for starting logs' collection based on provided k8s annotations' hints
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: [34427]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: [user]

receiver/receivercreator/README.md

@@ -456,9 +456,12 @@ receiver_creator/metrics:
      enabled: true
      # Define which receivers should be ignored when provided through annotations
      # ignore_receivers: []
+     # define which paths should be excluded from collection
+     # exclude_paths: []
 ```
 
-Find bellow the supported annotations that user can define to automatically enable receivers to start collecting metrics signals from the target Pods/containers.
+Find bellow the supported annotations that user can define to automatically enable receivers to start
+collecting metrics signals from the target Pods/containers.
 
 ### Supported metrics annotations
 
@@ -511,11 +514,91 @@ The current implementation relies on the implementation of `k8sobserver` extensi
 the [pod_endpoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.111.0/extension/observer/k8sobserver/pod_endpoint.go).
 The hints are evaluated per container by extracting the annotations from each [`Port` endpoint](#Port) that is emitted. 
 
+### Supported logs annotations
+
+This feature enables `filelog` receiver along with the `container` parser in order to collect logs from the discovered
+Pods.
+
+#### Enable/disable discovery
+
+`io.opentelemetry.discovery.logs/enabled` (example: `"true"`)
+
+By default `"false"`.
+
+In case users want to collect logs from all discovered containers by default, `default_logs_discovery` setting
+can be used.
+NOTE: it is recommended to exclude Collectors' self logs from default collection. Use `exclude_paths` to exclude them)
+
+The following configuration can be used:
+
+```yaml
+receiver_creator/logs:
+  watch_observers: [ k8s_observer ]
+  discovery:
+    enabled: true
+    # collect logs from all containers unless discovery is explicitly disabled
+    # for a container using the discovery hint
+    default_logs_discovery: true
+    # RECOMMENDED: ensure that Collectors' self logs are excluded to avoid circuiting logs
+    exclude_paths: [/var/log/pods/*opentelemetry-collector-agent*/opentelemetry-collector/*.log]
+```
+#### Define configuration
+The default configuration for the `filelog` receiver is the following:
+
+```yaml
+include:
+  - /var/log/pods/`pod.namespace`_`pod.name`_`pod.uid`/`container_name`/*.log
+include_file_name: false
+include_file_path: true
+operators:
+  - id: container-parser
+    type: container
+```
+This default can be extended using the respective annotation:
+`io.opentelemetry.discovery.logs/config`
+
+**Example:**
+
+```yaml
+io.opentelemetry.discovery.logs/config: |
+  include_file_name: true
+  max_log_size: "2MiB"
+  operators:
+    - type: regex_parser
+      regex: "^(?P<time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$"
+```
+
+Note that individual settings are overridden by the configuration provided by the hints while the operators list
+is extended keeping first the `container` parser.
+
+
+#### Support multiple target containers
+
+Users can target the annotation to a specific container by suffixing it with the name of that container:
+`io.opentelemetry.discovery.logs.<container_name>/endpoint`.
+For example:
+```yaml
+io.opentelemetry.discovery.logs.busybox/config: |
+  max_log_size: "3MiB"
+  operators:
+    - id: some
+      type: add
+      field: attributes.tag
+      value: hints
+```
+where `busybox` is the name of the target container.
+
+If a Pod is annotated with both container level hints and pod level hints the container level hints have priority and
+the Pod level hints are used as a fallback (see detailed example bellow).
+
+The current implementation relies on the implementation of `k8sobserver` extension and specifically
+the [pod_endpoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.111.0/extension/observer/k8sobserver/pod_endpoint.go).
+The hints are evaluated per container by extracting the annotations from each [`Pod Container` endpoint](#Pod Container) that is emitted.
 
 
 ### Examples
 
-#### Metrics example
+#### Metrics and Logs example
 
 Collector's configuration:
 ```yaml
@@ -525,12 +608,22 @@ receivers:
     discovery:
       enabled: true
     receivers:
+  
+  receiver_creator/logs:
+    watch_observers: [ k8s_observer ]
+    discovery:
+      enabled: true
+    receivers:
 
 service:
   extensions: [ k8s_observer]
   pipelines:
     metrics:
-      receivers: [ receiver_creator ]
+      receivers: [ receiver_creator/metrics ]
+      processors: []
+      exporters: [ debug ]
+    logs:
+      receivers: [ receiver_creator/logs ]
       processors: []
       exporters: [ debug ]
 ```
@@ -600,6 +693,21 @@ spec:
           endpoint: "http://`endpoint`/nginx_status"
           collection_interval: "30s"
           timeout: "20s"
+
+        # redis pod container logs hints
+        io.opentelemetry.discovery.logs.redis/enabled: "true"
+        io.opentelemetry.discovery.logs.redis/config: |
+          max_log_size: "4MiB"
+          operators:
+            - id: some
+              type: add
+              field: attributes.tag
+              value: logs_hints
+
+        # nginx pod container logs hints
+        io.opentelemetry.discovery.logs.webserver/enabled: "true"
+        io.opentelemetry.discovery.logs.webserver/config: |
+          max_log_size: "3MiB"
     spec:
       volumes:
         - name: nginx-conf

receiver/receivercreator/config.go

@@ -90,8 +90,10 @@ type Config struct {
 }
 
 type DiscoveryConfig struct {
-	Enabled         bool     `mapstructure:"enabled"`
-	IgnoreReceivers []string `mapstructure:"ignore_receivers"`
+	Enabled              bool     `mapstructure:"enabled"`
+	IgnoreReceivers      []string `mapstructure:"ignore_receivers"`
+	DefaultLogsDiscovery bool     `mapstructure:"default_logs_discovery"`
+	ExcludePaths         []string `mapstructure:"exclude_paths"`
 }
 
 func (cfg *Config) Unmarshal(componentParser *confmap.Conf) error {

receiver/receivercreator/config_test.go

@@ -299,3 +299,40 @@ func (*nopWithoutEndpointFactory) CreateTraces(
 		cfg:      cfg,
 	}, nil
 }
+
+type nopWithFilelogConfig struct {
+	Include         []string `mapstructure:"include"`
+	IncludeFileName bool     `mapstructure:"include_file_name"`
+	IncludeFilePath bool     `mapstructure:"include_file_path"`
+	Operators       []any    `mapstructure:"operators"`
+}
+
+type nopWithFilelogFactory struct {
+	rcvr.Factory
+}
+
+type nopWithFilelogReceiver struct {
+	mockComponent
+	consumer.Logs
+	consumer.Metrics
+	consumer.Traces
+	rcvr.Settings
+	cfg component.Config
+}
+
+func (*nopWithFilelogFactory) CreateDefaultConfig() component.Config {
+	return &nopWithFilelogConfig{}
+}
+
+func (*nopWithFilelogFactory) CreateLogs(
+	_ context.Context,
+	rcs rcvr.Settings,
+	cfg component.Config,
+	nextConsumer consumer.Logs,
+) (rcvr.Logs, error) {
+	return &nopWithEndpointReceiver{
+		Logs:     nextConsumer,
+		Settings: rcs,
+		cfg:      cfg,
+	}, nil
+}