fixing missing logs while pod log file being rolled over

[pszkamruk-splunk]

Description:
This PR contains changes related to the bug raised by one of our customer. They mentioned that not all logs were being collected from the log files due to the files being rotated.
The changes to fix above issue have been added by using featureGates.fixMissedLogsDuringLogRotation flag. Setting that flag to true will enable the fix and will allow connector to collect logs also from files which are being rolled over.

Testing: checked manually, no duplicated logs, no missed logs

[dmitryax]

@pszkamruk-splunk, have you verified that this actually solves the problem? Do you know what's the container runtime and k8s cluster the customer has? AFAIK containerd should be doing copy/truncate, in which case this change isn't needed, see: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/fileconsumer/design.md#rotated-files-that-end-up-out-of-the-matching-pattern. Also, there is relevant issue where it's ended up to be another problem

[pszkamruk-splunk]

@pszkamruk-splunk, have you verified that this actually solves the problem? Do you know what's the container runtime and k8s cluster the customer has? AFAIK containerd should be doing copy/truncate, in which case this change isn't needed, see: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/fileconsumer/design.md#rotated-files-that-end-up-out-of-the-matching-pattern. Also, there is relevant issue where it's ended up to be another problem

I tested it locally with loggen pods, and as shown in the attached picture we are collecting events from rotated files. If the fix is not enabled (featureGates.fixMissedLogsDuringLogRotation: false) then from time to time I was observing some dropped logs.
I saw docs regarding file rotation but I was not aware of that issue.
I asked support to confirm what container runtime is being used by customer.

My proposal is to proceed with this change and see if it will help on customer environment, and then we can decide if we want to keep it or remove it.

[pszkamruk-splunk]

@dmitryax
The agency runs OpenShift 4 - specifically v4.16 and at the time this ticket was opened v4.12. All versions of OpenShift 4.x leverage cri-o as the container runtime

[dmitryax]

My proposal is to proceed with this change and see if it will help on customer environment, and then we can decide if we want to keep it or remove it.

Sounds good to me. Please address my other comment above, and we will merge it