[filestream] - Filestream stops reading last few lines

[VihasMakwana]

There appears to be a race condition between the file watcher and harvester goroutines, when the file is modified right around the time the close.on_state_change.inactive timer expires:

File watcher goroutine:

• filestream receives a "write" event and updates internal state (fswatch.go).
• A harvester is already running at this stage, so it exits with a "harvester already running" error.

Harvester goroutine:

• filestream detects that the file has been idle for 5 minutes.
• It cancels the context before it could detect the new changes.

This leads to a situation where a write occurs, but the harvester exits too early due to the inactive timeout, and a new harvester isn't started because the watcher has already updates its state. As a result, new data may be missed.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[VihasMakwana]

Temporary mitigation is to increase the inactive value.

[belimawr]

This seems to be similar to #39732.

This is somehow a known issue, at least for me, I thought here was already an open issue describing it, but I could not find it, the closest I found was the one I liked above.

Because of this "event driven" architecture of the file watcher, if two or more events happen on the same scan, only one is handled, which can lead to the last few lines of a file being missed.

Reducing the scan interval (prospector.scanner.check_interval) is also another mitigation. The idea is that scanning more often, we will pick different events on different scans.

[belimawr]

Maybe we could start generating a list of events for each file on each scan 🤔