[Filebeat] Clean up Log input registry entries for removed inputs

[belimawr]

When migrating from Log input to Filestream, the registry entries from the Log input are never removed from the registry, even when the Log input is fully removed from the configuration. Keeping the entries in the registry is usually desirable, so users can add/edit/remove input configurations without losing the state of a file, however when all Log input instances are migrated to Filestream, the old entries are not needed any more.

When the registrar used by the Log input starts up, it reads all of its entries from the disk store and maintains an in-memory copy of them, during shut down all those entries are written to disk:

beats/filebeat/registrar/registrar.go

Lines 139 to 143 in d3be9bf

	defer func() {
	if err := writeStates(r.store, r.states.GetStates()); err != nil {
	r.log.Errorf("Error writing stopping registrar state to statestore: %v", err)
	}
	}()

This prevents Filestream's take over mode from truly removing registry entries from the Log input. Even though the take over mode removes them after the migration:

beats/filebeat/input/filestream/internal/input-logfile/store.go

Lines 481 to 487 in d3be9bf

	// "remove" from the disk store.
	// It will add a remove entry in the log file for this key, however
	// the Registrar used by the Log input will write to disk all states
	// it read when Filebeat was starting, thus "overriding" this delete.
	// We keep it here because when we remove the Log input we will ensure
	// the entry is actually remove from the disk store.
	s.store.persistentStore.Remove(k)

They get re-written during shutdown. This happens even if there are no active Log inputs because the registrar is started with Filebeat, even if no instances of the Log input are ever started. Registry entries from files that do not have an active input have their TTL set to -2.

This issue is about finding a way to clean up the disk store once we know the Log input entries are no longer needed.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)