filestream: track state for deleted files

[AndersonQ]

Summary

When new paths are added to a filestream input, Filebeat may re-ingest the full content of files it has previously processed but for which the state has been deleted. This happens because the state is tied to the file being tracked by filestream. Once the file is removed, its state is also removed via clean_removed: true (the current default). Thus, the file is treated as new if discovered under a different path after being marked as “deleted”.

A primary example is an existing Kubernetes integration being updated to ingest rotated log files, causing duplication for all previously rotated files. This issue aims to investigate and define a robust strategy to prevent this re-ingestion, ensuring a seamless experience when monitoring paths are updated.

It's similar to #43650.

Background

A primary example of this issue occurs when updating our Kubernetes integration:

1. Legacy Path: The original configuration monitored /var/log/containers/*, which contains symlinks to the active log file for each container.
2. Rotation: On log rotation, kubelet renames the active file (e.g., log.0 becomes log.0.timestamp). Filebeat, tracking the symlink, correctly registers this event as a file "deletion".
3. State Purge: With the default clean_removed: true setting, Filebeat removes the state for this "deleted" file from its registry to prevent indefinite growth.
4. Data Re-ingestion: When the configuration is updated to monitor /var/log/pods/*, Filebeat discovers the rotated files (e.g., log.0.timestamp). As there is no state associated with this specific path in its registry, it treats them as new files and ingests their entire content again.

Proposed Avenues for Investigation

We have identified two potential approaches to solve this issue.

1. Persist State for Removed Files (Preferred)

This is considered the most robust and promising solution.

Concept: The core idea is to modify filestream’s registry to retain the state of files that have been marked as deleted (e.g., they are moved from the monitored paths). Fingerprinting correctly identifies a file that was previously tracked, was marked as deleted, and appears later under a new path. If a file with a known fingerprint reappears, its previous state (offset) can be restored, and ingestion can resume from the correct position.

Challenges:

• A new, sophisticated state management strategy is required to allow state to be kept for deleted files and to eventually purge this historical state to prevent the indefinite growth of the registry.
• This approach is fundamentally incompatible with the legacy clean_inactive:0, legacy_clean_inactive:true behaviour. A solution for their coexistence must be found.
• Using clean_inactive instead of clean_removed is not a viable option. This approach's effectiveness depends on environment-specific tuning, and it's impossible to establish a one-size-fits-all default value that would work reliably. Furthermore, any proposed solution must be compatible with the default behaviour, which is clean_removed: true.

2. New ignore_older_for_path Configuration

This approach was considered as a potential mitigation but has significant drawbacks.

Concept: Introduce a new configuration directive where one can specify a path glob and a timestamp. Filebeat would ignore any file matching the glob if its creation time is before the given timestamp.

Drawbacks:

• Incomplete Solution: It only mitigates the issue. A small risk of re-ingestion remains for files that are rotated after the specified timestamp but before Filebeat restarts.
• Lacks Automation: There is currently no mechanism to dynamically set the current timestamp when generating an integration policy, making it incompatible with our requirement for a fully automatic, transparent user experience.

Requirements for Investigation

A successful solution must meet the following requirements:

Requirements

1. Automatic Prevention: The solution must automatically prevent the re-ingestion of previously processed files when new monitoring paths are added to a filestream input.
2. User Transparency for integrations: The mechanism must be entirely transparent under the Elastic-Agent (eg. Kubernetes integration), requiring no manual configuration or intervention.
3. Fingerprint-Based Identity: This state persistence mechanism must apply only to file identity fingerprint.
4. State Persistence and Updates: When a file is no longer on a monitored path, its fingerprint-keyed state must be retained in the registry for a configurable period and continue to be updated as events are acknowledged by the output.
5. State Restoration: If a file with a previously seen fingerprint for a deleted file is discovered (potentially at a new path), Filebeat must resume ingestion from the last known offset associated with that fingerprint.
6. Bounded State Store: A mechanism must exist to prevent the registry from growing indefinitely with historical fingerprint states.
7. Configurable State TTL: The solution should include a time-based eviction policy for states for deleted files. This would purge the state for a deleted file if no corresponding file has been seen for a configurable duration.
8. Performance: The mechanism for managing and loading states for removed files must ensure it's feasible to load the state into memory.

Acceptance Criteria

The primary use case driving this investigation is the Kubernetes integration upgrade. The solution will be considered successful if the following scenario is met:

• GIVEN an existing Kubernetes integration configured to monitor /var/log/containers/*.
• AND the cluster contains pods with log files that have already been rotated and now reside in /var/log/pods/*.
• WHEN the integration policy is updated to include the /var/log/pods/* path.
• THEN Filebeat must ingest only new data from both the active and rotated log files.
• AND the full content of the previously rotated log files must NOT be ingested again.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[belimawr]

Thanks @AndersonQ, the challenges and requirements look pretty clear
to me.

Some comments on semantics

I have some small comments on the semantics of a couple sentences:

Thus, the file is treated as new if discovered under a different
path after being marked as “deleted”.

If the state for a file does not exist (or has been deleted) and the
file re-appears under the paths Filestream is monitoring, it will be
treated as a new file (because it is a new file) regardless whether
the path is the same or not.

The core idea is to modify filestream’s registry to retain the state
of files that have been marked as deleted (e.g., they are moved from
the monitored paths).

The Filestram input does not "mark" files. It either has a state for
the file in the registry or not. By default when a file that is being
ingested is removed (it's not present in the monitored paths) its
registry entry is also removed.

Obs: Because of an implementation detail of Fiestream's store,
registry entries when removed are actually soft deleted (by setting
their TTL to zero), then when the store garbage collector runs, the
entries are actually removed from the registry.

Other use cases not considered

I can think of at least one use case that does not seem compatible
with this concept of keeping the state of deleted files for an set
period of time.

If an user wants files with the same content (or the same path), to be
ingested if they're new files. I recall working on a case
(unfortunately I'm not finding the case details) that had a requirement like this.
Their application
was some sort of cronjob, that would run periodically and create its
log file, every time the application was run, a new file with the same
content was created (the old log files were removed/rotated with a
reasonable policy). It was important to the user to have all of those
files ingested, even if they had exactly the same content.

If the application does not add timestamp to its log entries, it's
easy to image a situation like that generating files that would have
the same fingerprint. Even with the same content, at least the
ingestion time was different, making it useful to know every time that
specifically application was run and the status of the run. I do not
recall whether they were using Kubernetes or not.

This is the current behaviour of the Filestream input and the
Kubernets integration using it, changing this would be a breaking
change even though it is a very specific use case which clashes with
the acceptance criteria of this issue.

I'm fine with changing the behaviour, if necessary, to have defaults
that cover more cases for most users as long as we also give them an
option to keep the current behaviour.

[AndersonQ]

Thanks @AndersonQ, the challenges and requirements look pretty clear to me.

Some comments on semantics

I have some small comments on the semantics of a couple sentences:

Thus, the file is treated as new if discovered under a different
path after being marked as “deleted”.

If the state for a file does not exist (or has been deleted) and the file re-appears under the paths Filestream is monitoring, it will be treated as a new file (because it is a new file) regardless whether the path is the same or not.

The core idea is to modify filestream’s registry to retain the state
of files that have been marked as deleted (e.g., they are moved from
the monitored paths).

The Filestram input does not "mark" files. It either has a state for the file in the registry or not. By default when a file that is being ingested is removed (it's not present in the monitored paths) its registry entry is also removed.

Obs: Because of an implementation detail of Fiestream's store, registry entries when removed are actually soft deleted (by setting their TTL to zero), then when the store garbage collector runs, the entries are actually removed from the registry.

Other use cases not considered

I can think of at least one use case that does not seem compatible with this concept of keeping the state of deleted files for an set period of time.

If an user wants files with the same content (or the same path), to be ingested if they're new files. I recall working on a case (unfortunately I'm not finding the case details) that had a requirement like this. Their application was some sort of cronjob, that would run periodically and create its log file, every time the application was run, a new file with the same content was created (the old log files were removed/rotated with a reasonable policy). It was important to the user to have all of those files ingested, even if they had exactly the same content.

If the application does not add timestamp to its log entries, it's easy to image a situation like that generating files that would have the same fingerprint. Even with the same content, at least the ingestion time was different, making it useful to know every time that specifically application was run and the status of the run. I do not recall whether they were using Kubernetes or not.

This is the current behaviour of the Filestream input and the Kubernets integration using it, changing this would be a breaking change even though it is a very specific use case which clashes with the acceptance criteria of this issue.

I'm fine with changing the behaviour, if necessary, to have defaults that cover more cases for most users as long as we also give them an option to keep the current behaviour.

That's an interesting use-case.

what ever er go with I believe it needs some opt-in/out toggle. As long as we can automate it on the integrations, it's fine. It does not need to be enforced.

[cmacknz]

Keeping deleted state as a solution to this has some other obvious drawbacks that are not mentioned:

• It will cause the registry to grow to infinity without some cleanup strategy, that will then permanently delete the files, thus bringing this problem back.
• For it to work, a user has to deploy a version of filestream with this change and give it enough time to see the complete set of deleted files. Without both of these it does nothing and I think it's going to be very hard to get people to respect them both because they are not obvious at all.

Concept: Introduce a new configuration directive where one can specify a path glob and a timestamp. Filebeat would ignore any file matching the glob if its creation time is before the given timestamp.

Drawbacks:

Incomplete Solution: It only mitigates the issue. A small risk of re-ingestion remains for files that are rotated after the specified timestamp but before Filebeat restarts.

Lacks Automation: There is currently no mechanism to dynamically set the current timestamp when generating an integration policy, making it incompatible with our requirement for a fully automatic, transparent user experience.

Mitigating this is fine because it's a one time problem, especially if the implementation complexity is lower. It is annoying we don't have a good way to generate an absolute date in handlebars though googling around people have added helpers for it.

I still think the age of the files here is the key thing for solving this. We know the rotated files are always in the same directory as the "live" file and are always older than it and also appear deleted. So could we have a way to just ignore files meeting that specific criteria?

Like ignore_if_deleted: /var/log/containers/*.gz as a setting?

[AndersonQ]

It will cause the registry to grow to infinity without some cleanup strategy, that will then permanently delete the files, thus bringing this problem back.

That's why a clean up strategy is a requirement

Mitigating this is fine because it's a one time problem, especially if the implementation complexity is lower. It is annoying we don't have a good way to generate an absolute date in handlebars though googling around people have added helpers for it.

Yes, adding a helper is an option. Are you thinking about we doing the work or asking to the team that owns it to do it?

I still think the age of the files here is the key thing for solving this. We know the rotated files are always in the same directory as the "live" file and are always older than it and also appear deleted. So could we have a way to just ignore files meeting that specific criteria?

Like ignore_if_deleted: /var/log/containers/*.gz as a setting?

An entry for a file marked as deleted will eventually be removed, so we cannot assume it'll be there. If we can, than we don't even need a ignore_if_deleted: /var/log/containers/*.log.* (don't forget the most recent rotated log is plain text), we can just say if a a deleted file re-appear, we do not ingest it. We don't have state for deleted files, so the only alternative is ignoring the file completely.

The store GC/checkpoint is the issue, it can run at any time and actually remove the entries for deleted files. That's the main issue.

To have a single phase approach, either we go with

• some sort of ignore_older, fix the integration limitation and accept some data re-ingestion OR
• best effort for deleted files. We assume enough entries will be there, if they aren't, we re-ingest the files. We can ensure when filebeat starts it'll keep the entries for the deleted files "long enought", preventing the 1st GC from getting rid of it.

We could also try a hybrid approach, have both. A new ignore_rotated_older.path: /var/log/containers/*.log.*, ignore_rotated_older.timestamp: yyyy-mm-dd to try to account for the files without an entry on the registry and also do not re-ingest deleted files.

With a hybrid approach we could even allow the date to be in the future. That way we merge the 2 phased approach into one:

• wait until yyyy-mm-dd: any new file on this path is considered fully ingested
• after yyyy-mm-dd: resume normal tracking. The rotated files are still tracked, it operates as we want

[belimawr]

I like the "wait until" idea, setting it allows for any file in the new paths to be marked as fully ingested.

This "wait until" is somehow similar to something that crossed my mind during our meeting today. If Filestream can store some input state (instead of ingested files state), this would be from the input itself, based on its ID. We could have something like "new paths, ignore since first set". The way it would work:

• Filestream has this "input store" (it could be a simple json file per input), it does not need a fancy interface.
• The input store is empty (first deployment)
• Filestream sets in the store the current time alongside each glob in the new paths. If the store is not empty, we just read the value from it.
• Any file from matching the "new globs" is considered fully ingested if it was created before the timestamp stored for that glob.

This is somehow similar to our ignore_inactive: since_last_start, but only for a set of globs and with a specific timestamp that is set when Filestream first sees the setting being enabled.

The downside of this approach is that there is no way to clean up this "input store". Anyways, I'm focusing more on exploring the concept rather than the implementation details.

[rdner]

@belimawr @leehinman and I discussed an idea today.

If we switch to sqlite for the registry storage we can scale the on disk registry storage (currently operation journal) more efficiently. That would enable us to store information about all the files we ingested in the last e.g. 6 months (configurable). The information would primarily contain a file ID (fingerprint) and its last offset for each file Filebeat has ever touched in the configured TTL which can be measured in months.

Pros:

If we use the SQLite-based registry:

• We don't need most of the parameters related to registry cleanups, data retentions, etc. It's all simplified to a single TTL parameter.
• Every record is relatively fixed in size, there is no need to store the file path, only fingerprint, offset, TTL. We can predict the SQLite file size based on the amount of entries.
• We can track all the files we've touched months back in time. This will allow us to support cases like:
  • A file moved out of the ingestion path and then put back
  • A backup of a file gets restored (practically the same as above)
  • Switching ingestion paths to other directories that have duplicates of the files we've already ingested at some point
  • Migrating integrations (when involves path changes) without data duplications (for example the kubernetes integration)
• This would significantly simplify our registry implementation and would give more flexibility on how we can work with the data and what metrics we can collect on it.
• The forward migration to this architecture should be fairly straightforward.

Cons:

• Might cause a significant performance impact (needs to be measured working real outputs)
• Requires to use the fingerprint file identity in order for this architecture to work
• Requires us to deprecated a bunch of configuration parameters
• No feasible backwards migration for Filebeat, once the registry is converted to SQLite – we have to stick to it.

I think this should be coded as PoC and benchmarked with high throughput.

Unless @cmacknz can see some issues with this approach.

[belimawr]

I'd just add that the SQLite is a good option, but the core of the idea is to have a disk-based with good performance store.

The OtelCollector has a file storage extension that uses bbolt, it is worth looking into it as well.

[cmacknz]

Yes, our eventual destination is to be compatible with OpenTelemetry collector storage extensions, ideally the filestorage extension which currently uses bbolt.

Might cause a significant performance impact (needs to be measured working real outputs)

Beyond throughput the major thing to watch with databases will be any memory usage increase.

Requires us to deprecated a bunch of configuration parameters

Depending on what they are, we may not be able to get away with breaking changes just to do this.

No feasible backwards migration for Filebeat, once the registry is converted to SQLite – we have to stick to it

For Elastic Agent we'd be covered by the upgrade rollback feature keeping the old storage on disk to go back to. For Filebeat on its own, agreed we would need users to orchestrate this themselves.