Optimize message bus topic-matching logic by 100×

[twitu]

Pull Request

NautilusTrader prioritizes correctness and reliability, please follow existing patterns for validation and testing.

Summary

The message bus using pattern matching to identify subscribed topics to publish to. Message bus is on the hot path since each data item travels through it to respective strategies and components so performance and correctness is critical.

This PR adds regex based testing to validated matching logic with randomized testing. It also introduces an iterative algorithm with improves upon current implementation by 100x.

Using criterion shows that

• is_matching takes 2.5 us per match and scales linearly to 2.5 ms for 1000 topics
• regex takes 45 us to create dfa but scales almost constant time to 90 us for 1000 topics
• is_matching_backtracking does a whopping 25 ns per match and scales linearly to 25 us for 1000 topics

Backtracking can give very poor performance if the pattern is implemented poorly and as many wildcards. It also works well assuming that the pattern and topic length are small (<256 char). For 2000+ subscriptions with complex patterns consider adding a dynamic switch to regex based matching.

Type of change

• Performance improvement

Testing

Ensure new or changed logic is covered by tests.

• Affected code paths are already covered by the test suite
• I added/updated tests to cover new or changed logic

Add randomized testing for matching logic with existing regex implementation as reference.

[faysou]

It's impressive. When most things will be in rust it will be possible to optimise other things that become dominant in time.

[cjdsellers]

👌