-
Notifications
You must be signed in to change notification settings - Fork 475
test(profiling): make memalloc test less flaky [backport 4.1]
#15790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
bd2e597 to
089d923
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 246 ± 2 ms. The average import time from base is: 251 ± 2 ms. The import time difference between this PR and base is: -4.09 ± 0.09 ms. Import time breakdownThe following import paths have shrunk:
|
(cherry picked from commit ad2137d)
089d923 to
e04d2f7
Compare
Performance SLOsComparing candidate backport-15769-to-4.1 (e04d2f7) with baseline 4.1 (b22dec9) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.077µs (SLO: <20.000µs -9.6%) vs baseline: 📈 +20.9% Memory: ✅ 42.998MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.6% ✅ add_inplace_aspectTime: ✅ 15.011µs (SLO: <20.000µs 📉 -24.9%) vs baseline: +0.5% Memory: ✅ 42.841MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +4.2% ✅ add_inplace_noaspectTime: ✅ 0.341µs (SLO: <10.000µs 📉 -96.6%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ add_noaspectTime: ✅ 0.549µs (SLO: <10.000µs 📉 -94.5%) vs baseline: +1.1% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.6% ✅ bytearray_aspectTime: ✅ 18.041µs (SLO: <30.000µs 📉 -39.9%) vs baseline: -0.2% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.8% ✅ bytearray_extend_aspectTime: ✅ 23.748µs (SLO: <30.000µs 📉 -20.8%) vs baseline: ~same Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ bytearray_extend_noaspectTime: ✅ 2.679µs (SLO: <10.000µs 📉 -73.2%) vs baseline: +0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +3.9% ✅ bytearray_noaspectTime: ✅ 1.464µs (SLO: <10.000µs 📉 -85.4%) vs baseline: +0.6% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ bytes_aspectTime: ✅ 16.667µs (SLO: <20.000µs 📉 -16.7%) vs baseline: -0.8% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.5% ✅ bytes_noaspectTime: ✅ 1.403µs (SLO: <10.000µs 📉 -86.0%) vs baseline: +0.3% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytesio_aspectTime: ✅ 55.470µs (SLO: <70.000µs 📉 -20.8%) vs baseline: -0.1% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.6% ✅ bytesio_noaspectTime: ✅ 3.206µs (SLO: <10.000µs 📉 -67.9%) vs baseline: -0.2% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 14.666µs (SLO: <20.000µs 📉 -26.7%) vs baseline: -0.3% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ capitalize_noaspectTime: ✅ 2.555µs (SLO: <10.000µs 📉 -74.5%) vs baseline: -0.7% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.3% ✅ casefold_aspectTime: ✅ 14.583µs (SLO: <20.000µs 📉 -27.1%) vs baseline: -0.7% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.7% ✅ casefold_noaspectTime: ✅ 3.096µs (SLO: <10.000µs 📉 -69.0%) vs baseline: +0.1% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.1% ✅ decode_aspectTime: ✅ 15.561µs (SLO: <30.000µs 📉 -48.1%) vs baseline: +0.4% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.6% ✅ decode_noaspectTime: ✅ 1.612µs (SLO: <10.000µs 📉 -83.9%) vs baseline: +1.9% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 18.100µs (SLO: <30.000µs 📉 -39.7%) vs baseline: 📈 +22.4% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ encode_noaspectTime: ✅ 1.473µs (SLO: <10.000µs 📉 -85.3%) vs baseline: -0.5% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.2% ✅ format_aspectTime: ✅ 171.021µs (SLO: <200.000µs 📉 -14.5%) vs baseline: ~same Memory: ✅ 43.057MB (SLO: <43.250MB 🟡 -0.4%) vs baseline: +4.6% ✅ format_map_aspectTime: ✅ 192.122µs (SLO: <200.000µs -3.9%) vs baseline: +0.3% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ format_map_noaspectTime: ✅ 3.734µs (SLO: <10.000µs 📉 -62.7%) vs baseline: -0.7% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.1% ✅ format_noaspectTime: ✅ 3.174µs (SLO: <10.000µs 📉 -68.3%) vs baseline: +0.6% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.7% ✅ index_aspectTime: ✅ 15.191µs (SLO: <20.000µs 📉 -24.0%) vs baseline: -0.6% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.9% ✅ index_noaspectTime: ✅ 0.464µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.8% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 16.891µs (SLO: <20.000µs 📉 -15.5%) vs baseline: -0.2% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ join_noaspectTime: ✅ 1.484µs (SLO: <10.000µs 📉 -85.2%) vs baseline: -1.9% Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +4.7% ✅ ljust_aspectTime: ✅ 20.678µs (SLO: <30.000µs 📉 -31.1%) vs baseline: -0.7% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ ljust_noaspectTime: ✅ 2.671µs (SLO: <10.000µs 📉 -73.3%) vs baseline: +0.3% Memory: ✅ 42.900MB (SLO: <43.250MB 🟡 -0.8%) vs baseline: +4.7% ✅ lower_aspectTime: ✅ 17.876µs (SLO: <30.000µs 📉 -40.4%) vs baseline: +0.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ lower_noaspectTime: ✅ 2.407µs (SLO: <10.000µs 📉 -75.9%) vs baseline: +1.2% Memory: ✅ 42.900MB (SLO: <43.250MB 🟡 -0.8%) vs baseline: +4.7% ✅ lstrip_aspectTime: ✅ 17.726µs (SLO: <20.000µs 📉 -11.4%) vs baseline: +0.8% Memory: ✅ 43.057MB (SLO: <43.250MB 🟡 -0.4%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 1.809µs (SLO: <10.000µs 📉 -81.9%) vs baseline: -2.1% Memory: ✅ 42.880MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.4% ✅ modulo_aspectTime: ✅ 166.623µs (SLO: <200.000µs 📉 -16.7%) vs baseline: ~same Memory: ✅ 43.096MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +4.8% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 180.205µs (SLO: <200.000µs -9.9%) vs baseline: +2.0% Memory: ✅ 43.116MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +5.0% ✅ modulo_aspect_for_bytesTime: ✅ 168.405µs (SLO: <200.000µs 📉 -15.8%) vs baseline: ~same Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.332µs (SLO: <200.000µs 📉 -13.8%) vs baseline: +0.7% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.3% ✅ modulo_noaspectTime: ✅ 3.682µs (SLO: <10.000µs 📉 -63.2%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ replace_aspectTime: ✅ 214.049µs (SLO: <300.000µs 📉 -28.7%) vs baseline: -0.2% Memory: ✅ 43.037MB (SLO: <44.000MB -2.2%) vs baseline: +5.1% ✅ replace_noaspectTime: ✅ 5.197µs (SLO: <10.000µs 📉 -48.0%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ repr_aspectTime: ✅ 1.372µs (SLO: <10.000µs 📉 -86.3%) vs baseline: -0.7% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.5% ✅ repr_noaspectTime: ✅ 0.527µs (SLO: <10.000µs 📉 -94.7%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ rstrip_aspectTime: ✅ 19.120µs (SLO: <30.000µs 📉 -36.3%) vs baseline: -0.1% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ rstrip_noaspectTime: ✅ 1.905µs (SLO: <10.000µs 📉 -81.0%) vs baseline: +0.4% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ slice_aspectTime: ✅ 15.925µs (SLO: <20.000µs 📉 -20.4%) vs baseline: -0.2% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ slice_noaspectTime: ✅ 0.596µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 53.584µs (SLO: <80.000µs 📉 -33.0%) vs baseline: -0.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.6% ✅ stringio_noaspectTime: ✅ 3.546µs (SLO: <10.000µs 📉 -64.5%) vs baseline: +0.2% Memory: ✅ 42.880MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.4% ✅ strip_aspectTime: ✅ 17.536µs (SLO: <20.000µs 📉 -12.3%) vs baseline: -0.7% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ strip_noaspectTime: ✅ 1.822µs (SLO: <10.000µs 📉 -81.8%) vs baseline: +0.3% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ swapcase_aspectTime: ✅ 18.390µs (SLO: <30.000µs 📉 -38.7%) vs baseline: -0.2% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +6.0% ✅ swapcase_noaspectTime: ✅ 2.728µs (SLO: <10.000µs 📉 -72.7%) vs baseline: -1.5% Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.5% ✅ title_aspectTime: ✅ 18.100µs (SLO: <30.000µs 📉 -39.7%) vs baseline: -0.7% Memory: ✅ 42.900MB (SLO: <43.000MB 🟡 -0.2%) vs baseline: +4.6% ✅ title_noaspectTime: ✅ 2.663µs (SLO: <10.000µs 📉 -73.4%) vs baseline: +1.4% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ translate_aspectTime: ✅ 23.960µs (SLO: <30.000µs 📉 -20.1%) vs baseline: 📈 +17.2% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.7% ✅ translate_noaspectTime: ✅ 4.243µs (SLO: <10.000µs 📉 -57.6%) vs baseline: ~same Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.3% ✅ upper_aspectTime: ✅ 17.841µs (SLO: <30.000µs 📉 -40.5%) vs baseline: ~same Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ upper_noaspectTime: ✅ 2.395µs (SLO: <10.000µs 📉 -76.1%) vs baseline: +0.7% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.154µs (SLO: <10.000µs 📉 -48.5%) vs baseline: 📈 +22.0% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 4.299µs (SLO: <10.000µs 📉 -57.0%) vs baseline: -0.2% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.9% ✅ ospathjoin_aspectTime: ✅ 6.111µs (SLO: <10.000µs 📉 -38.9%) vs baseline: -0.4% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 6.224µs (SLO: <10.000µs 📉 -37.8%) vs baseline: +0.3% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +5.1% ✅ ospathnormcase_aspectTime: ✅ 3.511µs (SLO: <10.000µs 📉 -64.9%) vs baseline: +0.2% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 3.605µs (SLO: <10.000µs 📉 -64.0%) vs baseline: +0.5% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.1% ✅ ospathsplit_aspectTime: ✅ 4.865µs (SLO: <10.000µs 📉 -51.4%) vs baseline: +0.2% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +5.0% ✅ ospathsplit_noaspectTime: ✅ 4.957µs (SLO: <10.000µs 📉 -50.4%) vs baseline: ~same Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathsplitdrive_aspectTime: ✅ 3.770µs (SLO: <10.000µs 📉 -62.3%) vs baseline: +0.3% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.760µs (SLO: <10.000µs 📉 -92.4%) vs baseline: +0.5% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathsplitext_aspectTime: ✅ 4.641µs (SLO: <10.000µs 📉 -53.6%) vs baseline: +0.5% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathsplitext_noaspectTime: ✅ 4.699µs (SLO: <10.000µs 📉 -53.0%) vs baseline: -0.5% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +4.9% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
Backport ad2137d from #15769 to 4.1.
Description
This makes the
memalloctest less flaky by reducing the sampling interval. Currently,memallocis our top flaky test.The fix is:
heap_sample_sizelower to maximise the number of Samples we take (heap_sample_sizereally should beheap_sample_interval, since that's literally what it's named internally in the Collector, but that's an another-day-problem)assert_samplesparameter tosnapshot_and_parse_pprofto allow the caller not to do it. The reason is that we callsnapshot_and_parse_pprofonce just to reset the Sampler's state, so it's very possible that we didn't have Samples in the state at that moment (and it's completely OK if it happens). We only want to assert we have Samples when we want to... look at the Samples.Links