Refactor: Write all queue state to a single key on each write for reliable backup

[malus2077]

• This change modifies the persistence strategy to write all relevant queue information (start, stop, in-progress, size) together under a single key on every write operation.
• This ensures a more consistent and reliable backup of the queue's state, improving restoration accuracy after a potential crash.

Fixes #12890

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: malus2077 / name: Alan Lee (eb194f9, a431791)

[bogdandrutu]

Before starting to always write the size (there is a benefit that we never written it) we need to:

1. Decide if we also need to write the type of the size or we have 3 different sizes materialized on disk.
2. What to do when user changes the sizer type and we don’t know the size when loading metadata, the current implementation is a bit broken because it needs to have a full drain until size is restored.
3. Should we force full drain instead during shutdown or init without a known size?

[bogdandrutu]

Can you please do a PR that only adds the new proposed way (without the replacement or connection) then the second PR to start using that. As you saw there are some questions about this that I would like to address in the first PR, and also some questions about transition.

[malus2077]

Certainly, I can do that. I'm happy to proceed with the two-PR approach as you've outlined, allowing us to address your questions and concerns regarding the new method and the transition in a structured way.

[malus2077]

Can you please do a PR that only adds the new proposed way (without the replacement or connection) then the second PR to start using that. As you saw there are some questions about this that I would like to address in the first PR, and also some questions about transition.

The first PR has been created. #13067

[codecov]

Codecov Report

Attention: Patch coverage is 75.71802% with 93 lines in your changes missing coverage. Please review.

Project coverage is 91.36%. Comparing base (943627b) to head (a431791).
Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
...rterhelper/internal/queuebatch/persistent_queue.go	75.71%	69 Missing and 24 partials ⚠️

❌ Your patch check has failed because the patch coverage (75.71%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13047      +/-   ##
==========================================
- Coverage   91.53%   91.36%   -0.18%     
==========================================
  Files         504      504              
  Lines       28154    28384     +230     
==========================================
+ Hits        25772    25933     +161     
- Misses       1873     1923      +50     
- Partials      509      528      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bogdandrutu]

Are you planning to update this PR to use already merged code?

[bogdandrutu]

Use defer

[malus2077]

Are you planning to update this PR to use already merged code?

I'm currently finalizing the unit tests and getting ready to update the PR.

[malus2077]

Are you planning to update this PR to use already merged code?

The second PR has been created #13126 , PTAL.