JetStream recovery latency increases sub-linearly with backlog size for stalled durable consumers

[Abubakarsidiq01]

Description
While testing JetStream recovery behavior with file-backed storage and a stalled durable pull consumer, I observed that recovery latency after a server restart increases as publish backlog grows, though the scaling appears sub-linear. Recovery correctness is preserved, with no redelivery and accurate backlog tracking, but restart latency becomes noticeably longer with larger backlogs.
This may be expected behavior, but I wanted to share measured results since it could affect restart predictability for streams with large backlogs.

Environment

• NATS Server: 2.14.0-dev (current main)
• JetStream: enabled with file storage
• OS: Windows (PowerShell), reproducible locally
• Consumer type: durable pull consumer with explicit acknowledgments

Reproduction Steps

1. Start nats-server -js
2. Create a file-backed stream and a durable pull consumer with explicit acks
3. Publish messages while stalling the consumer by consuming one message and then stopping
4. Terminate the server during active publish
5. Restart the server
6. Measure time to first delivery after restart

Recovery time was measured as the time from server restart to successful delivery of the first message using Measure-Command { nats consumer next … }.

Observed Results
With approximately 100 published messages, recovery took about 2.77 seconds and resulted in 98 unprocessed messages.
With approximately 5000 published messages, recovery took about 4.80 seconds and resulted in 4,998 unprocessed messages.

In both cases, recovery completed successfully, no redelivered messages were observed, and unprocessed message counts matched the expected backlog. Recovery latency increased with backlog size, but not linearly.

Expected Behavior
Recovery should remain correct and deterministic, which it does. Ideally, restart latency would remain bounded or at least predictable as backlog size grows.

Notes:
These results suggest that recovery work scales with backlog size, though efficiently. I’m not sure if this behavior is expected or already well understood, but I wanted to share measured data in case it’s useful for understanding JetStream restart characteristics. I’m happy to dig further or help validate improvements or documentation if helpful

[wallyqs]

When restarting the server, make sure to stop it using the USR2 signal so that it gracefully stops, otherwise on each restart it will be reading all blocks to repair some state which is what I think is happening here (you would see a WRN in the logs on the stream).

[Abubakarsidiq01]

Thanks for the clarification. That makes sense. I was intentionally using an abrupt stop to observe crash-recovery behavior rather than graceful shutdown, but it’s helpful to understand that the repair path involves reading all blocks and explains the scaling I observed.
I can re-run the tests using USR2 to compare graceful vs crash restart behavior if that would be useful, or help document the expected differences between the two paths.