Description
Description: PR #13790 introduced an assumption that partition IDs map to specific stream config indices based on their modulo 10000 value. This change caused ingestion failures during testing - we use a custom ingestion plugin that use partition IDs outside the expected range (> 10000).
Current Behavior:
Partition IDs are assumed to follow a pattern where:
- 0-9999 maps to index 0
- 10000-19999 maps to index 1
- and so on
- This assumption is applied even when there is only one stream config
Problem: At Stripe, we use a custom ingestion plugin with partition IDs such as 65000 and 128000. These IDs are generated using a combination where:
The first half of the bytes represent a Kafka cluster
The latter half represent a Kafka partition ID
With the current implementation, a partition ID of 65000 incorrectly maps to index 5, despite having only one stream config in our table configuration.
Desired Behaviour:
- When there is only one stream config, the modulo 10000 calculation should not be used to determine the stream config index.
- The partition ID to stream config index mapping should be more flexible to accommodate custom ingestion plugins and partition ID schemes