Support complete file ingestion on update

[atoulme]

Component(s)

No response

Is your feature request related to a problem? Please describe.

We want to watch a file, and ingest all its contents on any modification of the file. The receiver should always keep the modification stamp of the file and compare it regularly to the file mstamp, or watch for file changes. Whenever this happens, the receiver should then read the entirety of the file and send it as logs.

Files watched would be configuration files of critical services, where all information must be logged for audit purposes.

Describe the solution you'd like

This is an exploratory issue that may result in a new component or an addition to an existing component.

The filelogreceiver may look like a possible approach to tackling this problem, however, it is not at all geared towards this use case. It is likely the stanza library has all the elements necessary to perform the work.

It is unclear if you actually need to persist modification timestamps - a first version can just assume on start that reading all files and send their contents is ok.

We should also set some sane limits, such that past a max size, the receiver should raise an error instead of accepting to read the file. That limit can be set by the user. A reasonable limit is 10K bytes.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/filelog: @djaglowski @andrzej-stencel. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[github-actions]

Pinging code owners for pkg/stanza: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[crobert-1]

Adding labels for related components with the goal of moving conversation along, not stating any opinion on my end.

[VihasMakwana]

We want to watch a file, and ingest all its contents on any modification of the file. The receiver should always keep the modification stamp of the file and compare it regularly to the file mstamp, or watch for file changes. Whenever this happens, the receiver should then read the entirety of the file and send it as logs.

If I understand it correctly, you want to send all contents of a file if it is modified?

$ cat file.txt
Hello there

• In this scenario, Hello there will be emitted.

$ cat file.txt
Hello there
Again, hello

• In this scenario, Hello there\nAgain, hello will be emitted.

is my understanding correct?

[bacherfl]

(Triage) removing needs-triage, and adding waiting-for-code-owners

[braydonk]

If we want to unilaterally reingest the entire file anyway, perhaps using something like fsnotify to detect would be sufficient rather than needing to re-jig any change detection that's in stanza right now?