Liveness Probe HTTP Endpoint

[blakerouse]

Describe the enhancement:

Currently the Elastic Agent running in a container does not have a liveness HTTP endpoint where kubernetes can check the overall health of the Elastic Agent container. This needs to be added so that in the case that the Elastic Agent is not working correctly it can be restarted by Kubernetes.

Some items that would be good for the liveness probe to be alerted to on failure:

• Not able to connect to Fleet Server (in managed mode)
• Overall bad state of inputs after a period of time

Liveness probe should have some subpaths defined for inputs that need to monitor there own liveness:

/liveness/endpoint - Checks if endpoint should be alive (see https://github.com/elastic/security-team/issues/3449#issuecomment-1112559420) for more details

Describe a specific use case for the enhancement or feature:

This needs to be added so that in the case that the Elastic Agent (or an integration that runs in a sidecar) is not working correctly it can be restarted by Kubernetes.

Additional Requirements

Enabling the liveness endpoint requires the ability to enable and possibly modify the agent HTTP configuration. This is currently not reloadable and cannot be configured from Fleet. For Fleet managed users to benefit from this we should make sure this can be turned on from Fleet.

elastic-agent/elastic-agent.yml

Lines 75 to 82 in b3e8275

	# http:
	# # enables http endpoint
	# enabled: false
	# # The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe.
	# # When using IP addresses, it is recommended to only use localhost.
	# host: localhost
	# # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled.
	# port: 6791

[ph]

@jlind23 @nimarezainia We should discuss this with our priority with @norrietaylor for the endpoint on k8s project.

[maneta]

Hey folks, we are also hitting an issue in the synthetics-service (https://github.com/elastic/synthetics-service/issues/435) where the agent fails in acquire the leader lease:

May 2, 2022 @ 14:56:46.498	elastic-agent-6qrjw	I0502 14:56:46.498250       7 leaderelection.go:278] failed to renew lease kube-system/elastic-agent-cluster-leader: timed out waiting for the condition

May 2, 2022 @ 14:56:46.498	elastic-agent-6qrjw	E0502 14:56:46.498350       7 leaderelection.go:301] Failed to release lock: resource name may not be empty

We think that this issue might be caused by the Apiserver being too slow to provide the lease. We are digging into metrics to certify that, but for now, the only solution we have to re-establish the agent is to manually restart the daemonsets. Would be great if we could set a liveness/readiness endpoint in the k8s template so k8s takes care of re-starting the pods for us.

For the deployment of agent itself we have crafted a helm chart(src but unfortunately no liveness/readiness is set(src)

Since we are reaching public beta phase, it would be really great if you could raise the priority on this is possible.

[rbrunan]

I would like to add to @maneta's comment that we would need an auto-recover mechanism for this leader lease loss. I mean, we should retry until the API is available again, to get the degraded metricset working again.

[joshdover]

@michel-laterman @pierrehilbert I know some pieces of this are blocked on v2 work being completed. Could we update the issue description with what's been completed and what's left to close out this issue?

[michel-laterman]

For v8.4.0 the fleet-gateway component is capable of setting the elastic-agent health to degraded if it fails 2 consecutive check ins. This will lead to cases where the health in fleet and the health from the local agent (through elastic-agent status) will differ.
The status command can now show the failing component or app in the message field, and also when the status has changed (through update_timestamp).

The liveness endpoint for v8.4.0 (GET /liveness) returns the health status (OK -> 200, DEGRADED/FAILURE -> 503), along with a json body that has

id # agent ID
status # status as a string
message # update message
update_timestamp # when the health status changed

No additional liveness endpoints are in v8.4.0.

For v8.5.0 it will likely need to be reimplemented in order to work with the new v5 architecture