Inputs.d usage in container environments is fuzzy

[ChrsMark]

Support for external configurations under inputs.d was introduced with 3c2072d . However the documentation around this is not quite helpful and it is only mentioned at https://www.elastic.co/guide/en/fleet/8.2/elastic-agent-configuration.html as you can put input configurations in YAML files into the folder {path.config}/inputs.d.

However this is not quite trivial to use this in container environments and even trying several different combinations didn't work.
I tried putting the inputs.d as /usr/share/elastic-agent/state/inputs.d, as /etc/elastic-agent when running with -c /etc/elastic-agent/elastic.yml and as /usr/share/elastic-agent/inputs.d but nothing works and logs don't provide any useful insight.

Note that Agent directory is under /usr/share/elastic-agent and it is started with -c /etc/elastic-agent/agent.yml.

cat container-paths.yml 
state_path: /usr/share/elastic-agent/state
config_path: /usr/share/elastic-agent/state

Find a complete k8s manifest below:
elastic-agent-standalone-kubernetes.yaml.txt

@ph could we have some help on clarifying those please?
cc: @gizas

This is important for #662.
Also related to #622.

[gizas]

Main manifest to install agent
elastic-agent-standalone-kubernetes_8.3 copy.txt

Extrnal Configs:
second_config_map copy.yml.txt

Extra Redis pod:
redis-pod copy.yaml.txt

Inside agent pod we can see files correctly mounted:

pwd && ls
/usr/share/elastic-agent/inputs.d
nats.yml  redis.yml

But still redis autodiscover is not working

[gizas]

cc @elastic/elastic-agent team if they have any input

[ChrsMark]

Note that it is not necessary to use redis.yml or any kind of autodiscovery here. The goal is to be able to put an input configuration under inputs.d directory and this to be honoured by Agent.

I have not tried this with Agent running as simple process on a native machine, however even in that case it's not fully clear where the inputs.d directory should be placed during the installation. When running Agent as a container it's again unclear where inputs.d should be placed. It would be nice if we could have docs with examples in each case for such a nice feature which from my perspective is quite hidden at the moment.

@cmacknz any chances of prioritising this for early 8.5 ? It should be a simple one, just validation and docs, for someone that actually knows/understands the codebase.

[ChrsMark]

Here is a minimal k8s manifest that I use to verify this:

elastic-agent-standalone.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic-agent-standalone-external-inputs
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
data:
  kubernetes.yml: |-
    inputs:
      - name: kubernetes-cluster-metrics
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.9.0
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: kubernetes.state_container
              type: metrics
            metricsets:
              - state_container
            add_metadata: true
            hosts:
              - 'kube-state-metrics:8080'
            period: 10s
            # If `https` is used to access `kube-state-metrics`, then to all `kubernetes.state_*` datasets should be added:
            # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            # ssl.certificate_authorities:
            #   - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-node-datastreams
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
data:
  agent.yml: |-
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
    agent:
      monitoring:
        enabled: true
        use_output: default
        logs: true
        metrics: true
    providers.kubernetes:
      node: ${NODE_NAME}
      scope: node
    inputs:
      - name: kubernetes-node-metrics
        type: kubernetes/metrics
        use_output: default
        meta:
          package:
            name: kubernetes
            version: 1.9.0
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: kubernetes.node
              type: metrics
            metricsets:
              - node
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            # On Openshift ssl configuration must be replaced:
            # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            # ssl.certificate_authorities:
            #   - /path/to/ca-bundle.crt
          - data_stream:
              dataset: kubernetes.pod
              type: metrics
            metricsets:
              - pod
            add_metadata: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            hosts:
              - 'https://${env.NODE_NAME}:10250'
            period: 10s
            ssl.verification_mode: none
            # On Openshift ssl configuration must be replaced:
            # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            # ssl.certificate_authorities:
            #   - /path/to/ca-bundle.crt
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent-standalone
  namespace: kube-system
  labels:
    app: elastic-agent-standalone
spec:
  selector:
    matchLabels:
      app: elastic-agent-standalone
  template:
    metadata:
      labels:
        app: elastic-agent-standalone
    spec:
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: elastic-agent-standalone
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
#      initContainers:
#        - name: myapp-container
#          image: busybox:1.28
#          command: ['sh', '-c', 'echo The app is running! && sleep 3600']
      containers:
        - name: elastic-agent-standalone
          image: docker.elastic.co/beats/elastic-agent:8.3.0
          args: [
            "-c", "/etc/elastic-agent/agent.yml",
            "-e",
          ]
          env:
            - name: ES_USERNAME
              value: "elastic"
            - name: ES_PASSWORD
              value: "oZFuK76eE6cUw4gfOybhmmw1"
            - name: ES_HOST
              value: "https://testfleet.es.us-central1.gcp.cloud.es.io:443"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 700Mi
            requests:
              cpu: 100m
              memory: 400Mi
          volumeMounts:
            - name: datastreams
              mountPath: /etc/elastic-agent/agent.yml
              readOnly: true
              subPath: agent.yml
            - name: external-inputs
              mountPath: /etc/elastic-agent/inputs.d
              readOnly: true
            - name: proc
              mountPath: /hostfs/proc
              readOnly: true
            - name: cgroup
              mountPath: /hostfs/sys/fs/cgroup
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: etc-kubernetes
              mountPath: /hostfs/etc/kubernetes
              readOnly: true
            - name: var-lib
              mountPath: /hostfs/var/lib
              readOnly: true
            - name: passwd
              mountPath: /hostfs/etc/passwd
              readOnly: true
            - name: group
              mountPath: /hostfs/etc/group
              readOnly: true
            - name: etcsysmd
              mountPath: /hostfs/etc/systemd
              readOnly: true
      volumes:
        - name: datastreams
          configMap:
            defaultMode: 0640
            name: agent-node-datastreams
        - name: external-inputs
          configMap:
            defaultMode: 0640
            name: elastic-agent-standalone-external-inputs
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: etc-kubernetes
          hostPath:
            path: /etc/kubernetes
        - name: var-lib
          hostPath:
            path: /var/lib
        - name: passwd
          hostPath:
            path: /etc/passwd
        - name: group
          hostPath:
            path: /etc/group
        - name: etcsysmd
          hostPath:
            path: /etc/systemd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent-standalone
subjects:
  - kind: ServiceAccount
    name: elastic-agent-standalone
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: elastic-agent-standalone
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-system
  name: elastic-agent-standalone
subjects:
  - kind: ServiceAccount
    name: elastic-agent-standalone
    namespace: kube-system
roleRef:
  kind: Role
  name: elastic-agent-standalone
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: elastic-agent-standalone-kubeadm-config
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: elastic-agent-standalone
    namespace: kube-system
roleRef:
  kind: Role
  name: elastic-agent-standalone-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent-standalone
  labels:
    k8s-app: elastic-agent-standalone
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
      - configmaps
      - serviceaccounts
      - persistentvolumes
      - persistentvolumeclaims
    verbs: ["get", "list", "watch"]
  # Enable this rule only if planing to use kubernetes_secrets provider
  #- apiGroups: [""]
  #  resources:
  #  - secrets
  #  verbs: ["get"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
      - daemonsets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  # required for apiserver
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources:
      - clusterrolebindings
      - clusterroles
      - rolebindings
      - roles
    verbs: ["get", "list", "watch"]
  - apiGroups: ["policy"]
    resources:
      - podsecuritypolicies
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent-standalone
  # should be the namespace where elastic-agent is running
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent-standalone-kubeadm-config
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
rules:
  - apiGroups: [""]
    resources:
      - configmaps
    resourceNames:
      - kubeadm-config
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent-standalone
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
---

Deploying this I expect that metrics are collected for node, pod and state_container metricsets/datasets. But only the first 2 are getting metrics which shows us that state_container which is defined under inputs.d is not enabled.

[ph]

Looking at the implementation, I think there is a problem with the way the inputs.d path is defined see.

// ExternalInputsPattern is a glob that matches the paths of external configuration files.
var ExternalInputsPattern = filepath.Join("inputs.d", "*.yml")

Which could mean the directory next to Elastic Agent, Due to upgrade support people are actually starting a symlink and not the actually binary. Let's me check the implementation, if should be a matter of wrapping the path with one of our helpers.

[ph]

@ChrsMark Can you test this PR #840 ?

[ChrsMark]

@ph I tested your PR locally and I still see don't see the external input having any impact 🤔 . Where the inputs.d directory should be mounted?

I do the following:

  - name: datastreams
    mountPath: /etc/elastic-agent/agent.yml
    readOnly: true
    subPath: agent.yml
  - name: external-inputs
    mountPath: /etc/elastic-agent/inputs.d

and Agent is started with -c /etc/elastic-agent/agent.yml. Is this correct? Any logs that would help to spot the issue here?

[blakerouse]

Maybe instead of guessing the path and making it relative to some other path we should make it a command line argument. The default can be /etc/elastic-agent/inputs.d and instead of adjusting based on the configuration path, we just give it its own parameter, that they need to set. That might make it clearer then it is at the moment.

[ChrsMark]

I think that would be great @blakerouse . Could it be part of #840 maybe?

[aleksmaus]

/etc/elastic-agent/inputs.d

Is there a use case that this file will be deployed in the location different from the Config()?

It might be useful to have a limited number agent files locations that the agent controls and can tighten up permissions

[ChrsMark]

@aleksmaus the permission concern is valid indeed, so it's up to you folks to decide this. As soon as the feature is documented and we have a straight-forward way to indicate users where to locate this additional directory I guess it is fine.

In the specific scenario with the k8s manifests we would like to locate this at /etc/elastic-agent/inputs.d as soon as it works, but if you have any other suggestion we can adopt :).