Windows Agents goes to unhealthy state on installation with system Integration.

[harshitgupta-qasource]

Kibana version: 8.4.3 BC2 Kibana cloud environment

Host OS: WINDOWS

Build details:
VERSION: 8.4.3 BC2
BUILD: 55572
COMMIT: 1ceb607762eaafa726c61d6eee5b95359142d4c4
Artifact link: https://staging.elastic.co/8.4.3-c9575cad/downloads/beats/elastic-agent/elastic-agent-8.4.3-windows-x86_64.zip

Preconditions:

1. 8.4.3 BC2 Kibana cloud environment should be available.
2. Windows agent v8.4.3 should be installed.

Steps to reproduce:

1. Navigate to Fleet> Create new Agent Policy .
2. Enroll windows agent with policy having system Integration.
3. Observe after 4-5 minutes agent went unhealthy.
4. Observe that no agent logs available under agent->logs tabs

Screenshot:

Logs:
elastic-agent-diagnostics-2022-09-29T05-04-57Z-00.zip

Expected Result:
Agent should come up with Healthy Status.

Additional Notes:

• This issue is not reproducible for MAC and Linux .tar agent.

[harshitgupta-qasource]

@manishgupta-qasource Please review

[manishgupta-qasource]

Secondary review for this ticket is Done

[amolnater-qasource]

FYI @jlind23

[jlind23]

@AndersonQ @fearful-symmetry @cmacknz can one of you take a look at this one? It seems to be a metricbeat_monitoring failure
{"log.level":"error","@timestamp":"2022-09-29T05:04:55.073Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":204},"message":"failed to dispatch actions, error: operator: failed to execute step sc-run, error: 2 errors occurred:\n\t* \"metricbeat_monitoring\" failed to prepare monitor for \"Metricbeat\": failed to create a directory \"\": mkdir : The system cannot find the path specified.\n\t* \"metricbeat_monitoring\" failed to prepare monitor for \"Metricbeat\": failed to create a directory \"\": mkdir : The system cannot find the path specified.\n\n: 2 errors occurred:\n\t* \"metricbeat_monitoring\" failed to prepare monitor for \"Metricbeat\": failed to create a directory \"\": mkdir : The system cannot find the path specified.\n\t* \"metricbeat_monitoring\" failed to prepare monitor for \"Metricbeat\": failed to create a directory \"\": mkdir : The system cannot find the path specified.\n\n","ecs.version":"1.6.0"}

[AndersonQ]

I'm looking at it

[AndersonQ]

hey, could you look at this issue #1361

Below are my findings, but I'm not sure about the expected behaviour, perhaps @michalpristas knows better.

The problem is here:

• monitorDrop returns "", then
• SidecarMonitor.Prepare calls os.MkdirAll("", 0775), which breaks.

I don't know what is the expected behaviour, what is the expected drop here :/

[michalpristas]

looking at it behavior is incorrect, code should check for drop != ""
only handle IO when it returns some normal path.
"" is a valid scenario for cases when metrics are not exposed using pipes/sockets

[michalpristas]

fix here: #1371
fix is a simple guard on drop path.
drop path can be empty if metrics endpoint is published over

• network (possible if configured in a spec file)
• NamedPipe (default on windows)

[michalpristas]

Fix backported to 8.4