fsync while writing last tombstone in purge

[tilakraj94]

This PR adds Sync() that ensures tombstones are durably persisted after stream purge operations. After purge completes, the message block and index files may become 0 bytes (whole directory moves to new dir). New changes might reflect partially in disk, In this state, the tombstone becomes the sole persistent record of the stream's last sequence number.

Without an explicit sync, the tombstone remains buffered in the OS page cache and is lost on power failure, leaving the stream unable to recover its sequence history and resulting in 0,0 state.

Signed-off-by: Tilak Raj [email protected]

[MauriceVanVeen]

Do you have a test or reproduction for this case? What is your setup, is this a single-node R1 stream or replicated?

Don't think the explicit sync is needed there though. If the setup is single-node R1 and you're more vulnerable to power failure, you'd probably want to configure sync: always in the server config instead. That would then already perform the sync as part of fs.writeTombstone itself.

[tilakraj9560]

Do you have a test or reproduction for this case? What is your setup, is this a single-node R1 stream or replicated?

Don't think the explicit sync is needed there though. If the setup is single-node R1 and you're more vulnerable to power failure, you'd probably want to configure sync: always in the server config instead. That would then already perform the sync as part of fs.writeTombstone itself.

Yes, this is a single R1 setup with the default sync interval. I’ll try to share a working example, but I’ve observed this behavior in older versions as well.

Using sync: always would address the issue, but that doesn’t mean state should be lost when sync: always is not enabled. The real problem occurs when the OS crashes immediately after a purge call has already returned success to the client (This only makes changes to OS Page Cache) unless sync: always. This case would lose OS caches, also why this happens with Purge, because whole directory moves to __msgs__ (older blks and index.db), so new directory/file changes (new blk and new index.db) could reflect partially.

This will not happen for replicated stream because of RAFT.

[tilakraj9560]

@MauriceVanVeen !lmb.syncAlways && fs.cfg.Replicas == 1 && lmb.mfd != nil I have added few conditions to the explicit sync.