Trosenblatt/async duration #297
Draft
+69
−24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds comprehensive timing metrics for scanner operations:
1. **I/O Duration Tracking**:
- Added `io_duration` field to `AsyncRuleInfo` to track time spent in async I/O operations
- Modified `process_async` to measure duration of async operations using `Instant::now()`
- `internal_scan` now aggregates I/O duration from all async jobs
2. **CPU Duration Histogram Metric**:
- Added `cpu_duration` histogram field to `ScannerMetrics` using highcard labels
- Metric name: `scanning.cpu_duration` (in nanoseconds)
- Calculated as: `total_duration - io_duration` to exclude I/O wait time
- Recorded in `internal_scan_with_metrics` after each scan
3. **New Return Types for Async Scan Methods**:
- Created `ScanMetrics` struct containing:
- `total_duration`: Total scan time
- `io_duration`: Time spent in I/O operations
- `num_async_rules`: Number of async rules executed
- Created `ScanResult` struct containing matches and metrics
- Updated `scan_async` and `scan_async_with_options` to return `ScanResult`
- Synchronous scan methods remain unchanged for backward compatibility
4. **Updated Tests**:
- Fixed async test assertions to work with new `ScanResult` return type
- All 293 tests passing
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds timing metrics to the
scan_asyncreturn value to enable better observability of scanning performance, particularly for tracking CPU time vs I/O time in async scans.Related JIRA: SDS-1904
Changes
Core Changes in dd-sds
New
ScanMetricsstruct - Captures timing information:total_duration: Total time from scan start to completionio_duration: Time spent in I/O operations (async rules making network requests)num_async_rules: Count of async rules executedNew
ScanResultstruct - Wraps scan results with metrics:matches: Vector ofRuleMatchobjects (existing functionality)metrics:ScanMetricswith timing informationUpdated
scan_asyncreturn type:Result<Vec<RuleMatch>, ScannerError>Result<ScanResult, ScannerError>I/O Duration Tracking:
total_duration - io_durationBreaking Changes
scan_async()andscan_async_with_options()now returnScanResultinstead ofVec<RuleMatch>result.matchesinstead of using the result directlyUsage in sds-shared-library
This change enables the sds-shared-library JNI bindings to report a new metric:
scan_cpu_duration_nsFor Synchronous Scans:
For Asynchronous Scans:
(completion_time - start_time) - io_durationThis provides accurate CPU usage metrics for both sync and async scanning patterns, enabling better performance analysis and resource planning.
Testing
ScanResultstructure