[no-release-notes] archive conjoin logic #9539
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added conjoinAll method to archiveWriter that takes multiple archiveReader instances and combines them into a single archive. The method currently returns a stub implementation with proper validation and error handling. Added comprehensive test that validates the expected behavior: - Creates two test archives with different chunk prefixes - Tests that combined reader contains all chunks from both archives - Verifies data integrity is maintained through the conjunction process - Includes helper function to reduce test code duplication 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…ence - Add full implementation of conjoinAll method that merges multiple archive readers - Includes context parameter, data span sorting, and memory-efficient io.Copy - Uses indexFinalizeFlushArchive for proper archive finalization and file persistence - Fix createTestArchive to properly compress chunk data before writing byte spans - Remove redundant readerAtAdapter type in favor of existing readerAtWithStatsBridge - All tests pass including new TestArchiveConjoinAll validation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Change conjoinAll return type from (archiveReader, error) to just error - Replace indexFinalizeFlushArchive with indexFinalize for in-memory completion - Remove file persistence logic to make method suitable for unit testing - Update test to create archive reader from in-memory bytes using provided pattern - Add documentation clarifying the method completes archive writing in memory only This makes conjoinAll more appropriate for unit tests that don't need disk materialization. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…ure duplicate handling - Add TestArchiveConjoinAllDuplicateChunk test that expects successful deduplication - Create createTestArchiveWithHashes helper for precise hash control - Create createArchiveWithDuplicates helper for mixed duplicate/unique scenarios - Test verifies combined archive contains all expected chunks after deduplication - Currently fails as expected since conjoinAll errors on duplicates - Test is correctly structured for future implementation of duplicate handling The helper functions will enable easy expansion to 10+ archives with complex duplicate patterns. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Update conjoinAll to allow duplicate chunks instead of erroring - Each chunk gets its own index entry pointing to its actual data location - Enhanced TestArchiveConjoinAllDuplicateChunk with more comprehensive test cases - Validate that duplicate chunks appear twice in the index as expected - Maintain performance by writing entire data blocks from each archive 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add reconstructHashFromPrefixAndSuffix helper function in archive_reader.go - Update archive_writer.go to use the helper function - Remove duplicate hash reconstruction logic while maintaining functionality - All tests continue to pass 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Refactor createTestArchive to delegate to createTestArchiveWithHashes - Remove ~35 lines of duplicate archive creation code - Maintain same functionality with cleaner test helper hierarchy - All tests continue to pass 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Enhanced createTestArchiveWithHashes to support mixed compression types and consolidated duplicate archive creation code. Fixed variable shadowing issue where 'chunks' parameter was shadowing the chunks package name. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Added TestArchiveConjoinAllComprehensive that tests complex scenarios: - 10 initial archives with mixed compression and chunk sizes (10-250 bytes) - Shared chunks duplicated across archives with mixed Snappy/zStd compression - Nested conjoin operations (first conjoin 10 archives, then conjoin result with 3 more) - Single validation loop testing all expected chunks and hashes - Proper buffer sizing to handle large amounts of data 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…n verification Added precise chunk count verification and comprehensive iteration testing: - Replaced approximate assertions with exact chunk count verification - Added iteration testing to verify each chunk ref is visited exactly once - Verify duplicate chunks appear the correct number of times during iteration - Ensure total iteration count equals chunk count for complete validation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Contributor
|
@coffeegoddd DOLT
|
Contributor
|
@macneale4 DOLT
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for the business logic of conjoining archives. It is not accessible in any way by a user at this point. Just implementation and unit tests using in memory buffers. Materializing to disk and integrating with the GC process is still required.