Skip to content

Conversation

@MahdiBM
Copy link
Contributor

@MahdiBM MahdiBM commented Jul 15, 2025

Description

Only the last commit is related to this PR. The other commits are PR [1/4] and [2/4]. We'll have to merge these in order.

Partially resolves #327.

Adds support for min/max "range"s and base/tolerancePercentage "relative"s in static threshold files.

Waiting on your feedback first. When changes are settled I can add the docs as well.

Feel free to review on your own schedule.

EDIT: now that things are somewhat stable, i notice the next PR ([4/4]) has made a decent amount of changes to this PR's changes.
I could try to rebase things and all but that'll be a bit of trouble. Or we could skip this PR and review the next one which also includes this PR but with more changes. Not sure.

How Has This Been Tested?

Manually in my PRs. I'll be happy to add unit tests where you see appropriate as well.

Minimal checklist:

  • I have performed a self-review of my own code
  • I have added DocC code-level documentation for any public interfaces exported by the package
  • I have added unit and/or integration tests that prove my fix is effective or that my feature works

@MahdiBM MahdiBM changed the title Mmbm range relative thresholds feat(minor): Support range and relative in static threshold files [3/4] Jul 15, 2025
@codecov
Copy link

codecov bot commented Jul 15, 2025

Codecov Report

Attention: Patch coverage is 62.33463% with 484 lines in your changes missing coverage. Please review.

Project coverage is 66.00%. Comparing base (3db567f) to head (04291aa).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
Sources/Benchmark/BenchmarkThresholds.swift 0.77% 129 Missing ⚠️
Sources/Benchmark/BenchmarkResult.swift 52.91% 81 Missing ⚠️
Sources/Benchmark/BenchmarkMetric+Defaults.swift 44.71% 47 Missing ⚠️
Sources/Benchmark/Benchmark.swift 44.26% 34 Missing ⚠️
...ark/MallocStats/MallocStatsProducer+jemalloc.swift 52.78% 34 Missing ⚠️
.../Benchmark/Benchmark+ConvenienceInitializers.swift 0.00% 32 Missing ⚠️
...urces/Benchmark/BenchmarkThresholds+Defaults.swift 21.95% 32 Missing ⚠️
Sources/Benchmark/OutputSuppressor.swift 0.00% 26 Missing ⚠️
Sources/Benchmark/BenchmarkRunner.swift 57.14% 21 Missing ⚠️
Sources/Benchmark/BenchmarkExecutor.swift 82.14% 15 Missing ⚠️
... and 7 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #330      +/-   ##
==========================================
- Coverage   69.48%   66.00%   -3.48%     
==========================================
  Files          33       33              
  Lines        3938     4276     +338     
==========================================
+ Hits         2736     2822      +86     
- Misses       1202     1454     +252     
Files with missing lines Coverage Δ
Sources/Benchmark/ARCStats/ARCStats.swift 100.00% <100.00%> (ø)
Sources/Benchmark/ARCStats/ARCStatsProducer.swift 85.29% <100.00%> (+0.92%) ⬆️
Sources/Benchmark/BenchmarkClock.swift 30.14% <100.00%> (-3.58%) ⬇️
Sources/Benchmark/Blackhole.swift 25.00% <ø> (ø)
Sources/Benchmark/MallocStats/MallocStats.swift 100.00% <100.00%> (ø)
Sources/Benchmark/Progress/ProgressElements.swift 92.54% <100.00%> (+0.11%) ⬆️
Tests/BenchmarkTests/BenchmarkMetricsTests.swift 97.75% <100.00%> (-1.12%) ⬇️
Tests/BenchmarkTests/BenchmarkResultTests.swift 100.00% <100.00%> (ø)
Tests/BenchmarkTests/BenchmarkRunnerTests.swift 92.68% <100.00%> (+0.38%) ⬆️
Tests/BenchmarkTests/BenchmarkTests.swift 97.37% <100.00%> (+0.31%) ⬆️
... and 19 more
Files with missing lines Coverage Δ
Sources/Benchmark/ARCStats/ARCStats.swift 100.00% <100.00%> (ø)
Sources/Benchmark/ARCStats/ARCStatsProducer.swift 85.29% <100.00%> (+0.92%) ⬆️
Sources/Benchmark/BenchmarkClock.swift 30.14% <100.00%> (-3.58%) ⬇️
Sources/Benchmark/Blackhole.swift 25.00% <ø> (ø)
Sources/Benchmark/MallocStats/MallocStats.swift 100.00% <100.00%> (ø)
Sources/Benchmark/Progress/ProgressElements.swift 92.54% <100.00%> (+0.11%) ⬆️
Tests/BenchmarkTests/BenchmarkMetricsTests.swift 97.75% <100.00%> (-1.12%) ⬇️
Tests/BenchmarkTests/BenchmarkResultTests.swift 100.00% <100.00%> (ø)
Tests/BenchmarkTests/BenchmarkRunnerTests.swift 92.68% <100.00%> (+0.38%) ⬆️
Tests/BenchmarkTests/BenchmarkTests.swift 97.37% <100.00%> (+0.31%) ⬆️
... and 19 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc7e44d...04291aa. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +99 to +102
precondition(
self.containsAnyValue,
"RelativeOrRange must contain either a relative or range, but contains neither"
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the next PR I've relaxed all these preconditions and made them just print warnings.
I've noticed the benchmark-tool running doesn't work too well with preconditions. It'll be hard to debug them.

if let threshold = thresholdsFromCode.absolute[percentile],
!(-threshold...threshold).contains(absoluteDifference)
{
let deviation = ThresholdDeviation(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure about these SwiftLint errors/warnings. Some of them doesn't play well with swift-format, like this one.

Let me know what I should do.

Comment on lines +165 to +173
debugDescription: """
RelativeOrRange thresholds object does not contain either a valid relative or range.
For relative thresholds, both 'base' (Int) and 'tolerancePercentage' (Double) must be present and valid.
For range thresholds, both 'min' (Int) and 'max' (Int) must be present and valid.
You can declare both relative and range in the same object together, or just one of them.
Example: { "min": 90, "max": 110 }
Example: { "base": 115, "tolerancePercentage": 5.5 }
Example: { "base": 115, "tolerancePercentage": 4.5, "min": 90, "max": 110 }
"""
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some info.

This is also backward-compatible. It'll try to read an Int first and if it fails then it tries to read the new RelativeOrRange.

@MahdiBM MahdiBM force-pushed the mmbm-range-relative-thresholds branch from 04291aa to f441e9e Compare August 26, 2025 16:22
@MahdiBM MahdiBM force-pushed the mmbm-range-relative-thresholds branch from 998e8b7 to 3137c2c Compare August 26, 2025 16:26
@MahdiBM
Copy link
Contributor Author

MahdiBM commented Aug 26, 2025

@hassila tried to properly rebase these 2 remaining PRs.
As mentioned, this PR has had some changes/improvements in the next PR.
I'm not sure what you prefer, i can do a self-review on this PR if you want to merge this first, or on the next PR if you want to completely skip this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Support a Range as threshold-tolerance

1 participant