-
Notifications
You must be signed in to change notification settings - Fork 475
chore(ci): update system-tests .github SHA #15762
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
Co-authored-by: Munir Abdinur <[email protected]>
Performance SLOsComparing candidate bump-system-tests (02245d6) with baseline main (f748494) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 17.906µs (SLO: <20.000µs 📉 -10.5%) vs baseline: 📈 +21.1% Memory: ✅ 42.920MB (SLO: <43.250MB 🟡 -0.8%) vs baseline: +4.8% ✅ add_inplace_aspectTime: ✅ 14.958µs (SLO: <20.000µs 📉 -25.2%) vs baseline: ~same Memory: ✅ 42.920MB (SLO: <43.250MB 🟡 -0.8%) vs baseline: +4.7% ✅ add_inplace_noaspectTime: ✅ 0.336µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +0.1% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ add_noaspectTime: ✅ 0.549µs (SLO: <10.000µs 📉 -94.5%) vs baseline: +0.7% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytearray_aspectTime: ✅ 18.042µs (SLO: <30.000µs 📉 -39.9%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ bytearray_extend_aspectTime: ✅ 24.110µs (SLO: <30.000µs 📉 -19.6%) vs baseline: +0.9% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ bytearray_extend_noaspectTime: ✅ 2.744µs (SLO: <10.000µs 📉 -72.6%) vs baseline: ~same Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ bytearray_noaspectTime: ✅ 1.464µs (SLO: <10.000µs 📉 -85.4%) vs baseline: -0.9% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ bytes_aspectTime: ✅ 16.677µs (SLO: <20.000µs 📉 -16.6%) vs baseline: -0.2% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.2% ✅ bytes_noaspectTime: ✅ 1.425µs (SLO: <10.000µs 📉 -85.8%) vs baseline: +0.8% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytesio_aspectTime: ✅ 55.351µs (SLO: <70.000µs 📉 -20.9%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ bytesio_noaspectTime: ✅ 3.278µs (SLO: <10.000µs 📉 -67.2%) vs baseline: -0.1% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ capitalize_aspectTime: ✅ 14.671µs (SLO: <20.000µs 📉 -26.6%) vs baseline: +0.1% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ capitalize_noaspectTime: ✅ 2.617µs (SLO: <10.000µs 📉 -73.8%) vs baseline: +0.6% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ casefold_aspectTime: ✅ 14.665µs (SLO: <20.000µs 📉 -26.7%) vs baseline: +0.1% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ casefold_noaspectTime: ✅ 3.145µs (SLO: <10.000µs 📉 -68.5%) vs baseline: -1.3% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ decode_aspectTime: ✅ 15.609µs (SLO: <30.000µs 📉 -48.0%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ decode_noaspectTime: ✅ 1.603µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.8% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 18.190µs (SLO: <30.000µs 📉 -39.4%) vs baseline: 📈 +22.7% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ encode_noaspectTime: ✅ 1.504µs (SLO: <10.000µs 📉 -85.0%) vs baseline: -0.2% Memory: ✅ 42.900MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.6% ✅ format_aspectTime: ✅ 172.236µs (SLO: <200.000µs 📉 -13.9%) vs baseline: +0.8% Memory: ✅ 43.273MB (SLO: <43.250MB ~same) vs baseline: +5.2% ✅ format_map_aspectTime: ✅ 190.922µs (SLO: <200.000µs -4.5%) vs baseline: -0.1% Memory: ✅ 43.155MB (SLO: <43.500MB 🟡 -0.8%) vs baseline: +4.9% ✅ format_map_noaspectTime: ✅ 3.832µs (SLO: <10.000µs 📉 -61.7%) vs baseline: +1.0% Memory: ✅ 42.998MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 3.143µs (SLO: <10.000µs 📉 -68.6%) vs baseline: -1.0% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.8% ✅ index_aspectTime: ✅ 15.241µs (SLO: <20.000µs 📉 -23.8%) vs baseline: -0.6% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ index_noaspectTime: ✅ 0.464µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.6% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 17.060µs (SLO: <20.000µs 📉 -14.7%) vs baseline: +0.1% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ join_noaspectTime: ✅ 1.548µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -0.5% Memory: ✅ 42.998MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +5.0% ✅ ljust_aspectTime: ✅ 20.687µs (SLO: <30.000µs 📉 -31.0%) vs baseline: -0.3% Memory: ✅ 42.998MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.7% ✅ ljust_noaspectTime: ✅ 2.703µs (SLO: <10.000µs 📉 -73.0%) vs baseline: +0.4% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.8% ✅ lower_aspectTime: ✅ 17.899µs (SLO: <30.000µs 📉 -40.3%) vs baseline: -0.2% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.1% ✅ lower_noaspectTime: ✅ 2.421µs (SLO: <10.000µs 📉 -75.8%) vs baseline: -0.2% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ lstrip_aspectTime: ✅ 17.654µs (SLO: <20.000µs 📉 -11.7%) vs baseline: +0.7% Memory: ✅ 42.979MB (SLO: <43.250MB 🟡 -0.6%) vs baseline: +4.8% ✅ lstrip_noaspectTime: ✅ 1.858µs (SLO: <10.000µs 📉 -81.4%) vs baseline: -0.5% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 166.559µs (SLO: <200.000µs 📉 -16.7%) vs baseline: +0.3% Memory: ✅ 43.096MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +4.8% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 179.849µs (SLO: <200.000µs 📉 -10.1%) vs baseline: +2.9% Memory: ✅ 43.195MB (SLO: <43.500MB 🟡 -0.7%) vs baseline: +5.1% ✅ modulo_aspect_for_bytesTime: ✅ 168.486µs (SLO: <200.000µs 📉 -15.8%) vs baseline: ~same Memory: ✅ 43.195MB (SLO: <43.500MB 🟡 -0.7%) vs baseline: +4.8% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.182µs (SLO: <200.000µs 📉 -13.9%) vs baseline: +0.1% Memory: ✅ 43.155MB (SLO: <43.500MB 🟡 -0.8%) vs baseline: +5.0% ✅ modulo_noaspectTime: ✅ 3.670µs (SLO: <10.000µs 📉 -63.3%) vs baseline: -0.4% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ replace_aspectTime: ✅ 212.141µs (SLO: <300.000µs 📉 -29.3%) vs baseline: ~same Memory: ✅ 43.175MB (SLO: <44.000MB 🟡 -1.9%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 2.907µs (SLO: <10.000µs 📉 -70.9%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ repr_aspectTime: ✅ 1.436µs (SLO: <10.000µs 📉 -85.6%) vs baseline: +1.4% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ repr_noaspectTime: ✅ 0.524µs (SLO: <10.000µs 📉 -94.8%) vs baseline: +0.2% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ rstrip_aspectTime: ✅ 19.027µs (SLO: <30.000µs 📉 -36.6%) vs baseline: +0.6% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ rstrip_noaspectTime: ✅ 2.017µs (SLO: <10.000µs 📉 -79.8%) vs baseline: +3.8% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 15.859µs (SLO: <20.000µs 📉 -20.7%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 0.597µs (SLO: <10.000µs 📉 -94.0%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 53.866µs (SLO: <80.000µs 📉 -32.7%) vs baseline: -0.2% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ stringio_noaspectTime: ✅ 3.628µs (SLO: <10.000µs 📉 -63.7%) vs baseline: -0.7% Memory: ✅ 43.057MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.1% ✅ strip_aspectTime: ✅ 17.663µs (SLO: <20.000µs 📉 -11.7%) vs baseline: +0.2% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 1.884µs (SLO: <10.000µs 📉 -81.2%) vs baseline: +1.9% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ swapcase_aspectTime: ✅ 18.458µs (SLO: <30.000µs 📉 -38.5%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ swapcase_noaspectTime: ✅ 2.791µs (SLO: <10.000µs 📉 -72.1%) vs baseline: -0.8% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 18.270µs (SLO: <30.000µs 📉 -39.1%) vs baseline: ~same Memory: ✅ 43.077MB (SLO: <43.000MB +0.2%) vs baseline: +5.0% ✅ title_noaspectTime: ✅ 2.693µs (SLO: <10.000µs 📉 -73.1%) vs baseline: +0.7% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 24.413µs (SLO: <30.000µs 📉 -18.6%) vs baseline: 📈 +18.9% Memory: ✅ 42.959MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ translate_noaspectTime: ✅ 4.346µs (SLO: <10.000µs 📉 -56.5%) vs baseline: ~same Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ upper_aspectTime: ✅ 17.955µs (SLO: <30.000µs 📉 -40.1%) vs baseline: +0.9% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ upper_noaspectTime: ✅ 2.417µs (SLO: <10.000µs 📉 -75.8%) vs baseline: -0.7% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.190µs (SLO: <10.000µs 📉 -48.1%) vs baseline: 📈 +21.3% Memory: ✅ 41.484MB (SLO: <43.500MB -4.6%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 4.296µs (SLO: <10.000µs 📉 -57.0%) vs baseline: -0.7% Memory: ✅ 41.425MB (SLO: <43.500MB -4.8%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 6.279µs (SLO: <10.000µs 📉 -37.2%) vs baseline: ~same Memory: ✅ 41.327MB (SLO: <43.500MB -5.0%) vs baseline: +4.7% ✅ ospathjoin_noaspectTime: ✅ 6.338µs (SLO: <10.000µs 📉 -36.6%) vs baseline: +0.5% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 3.561µs (SLO: <10.000µs 📉 -64.4%) vs baseline: +0.4% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.7% ✅ ospathnormcase_noaspectTime: ✅ 3.622µs (SLO: <10.000µs 📉 -63.8%) vs baseline: -1.2% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.8% ✅ ospathsplit_aspectTime: ✅ 4.904µs (SLO: <10.000µs 📉 -51.0%) vs baseline: -0.3% Memory: ✅ 41.543MB (SLO: <43.500MB -4.5%) vs baseline: +5.2% ✅ ospathsplit_noaspectTime: ✅ 5.020µs (SLO: <10.000µs 📉 -49.8%) vs baseline: +0.2% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplitdrive_aspectTime: ✅ 3.789µs (SLO: <10.000µs 📉 -62.1%) vs baseline: +0.7% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.747µs (SLO: <10.000µs 📉 -92.5%) vs baseline: ~same Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +5.0% ✅ ospathsplitext_aspectTime: ✅ 4.670µs (SLO: <10.000µs 📉 -53.3%) vs baseline: +1.3% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 4.667µs (SLO: <10.000µs 📉 -53.3%) vs baseline: -0.3% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.367µs (SLO: <20.000µs 📉 -83.2%) vs baseline: 📈 +13.7% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-count-metrics-100-timesTime: ✅ 199.931µs (SLO: <220.000µs -9.1%) vs baseline: ~same Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.310µs (SLO: <20.000µs 📉 -83.5%) vs baseline: +0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 1-distribution-metrics-100-timesTime: ✅ 213.167µs (SLO: <230.000µs -7.3%) vs baseline: -0.2% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ 1-gauge-metric-1-timesTime: ✅ 2.173µs (SLO: <20.000µs 📉 -89.1%) vs baseline: +0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.947µs (SLO: <150.000µs -8.7%) vs baseline: +0.7% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.6% ✅ 1-rate-metric-1-timesTime: ✅ 3.104µs (SLO: <20.000µs 📉 -84.5%) vs baseline: +0.5% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-rate-metrics-100-timesTime: ✅ 212.854µs (SLO: <250.000µs 📉 -14.9%) vs baseline: -0.5% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 100-count-metrics-100-timesTime: ✅ 19.950ms (SLO: <22.000ms -9.3%) vs baseline: -0.9% Memory: ✅ 34.957MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.260ms (SLO: <2.550ms 📉 -11.4%) vs baseline: +1.5% Memory: ✅ 35.173MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.8% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.411ms (SLO: <1.550ms -9.0%) vs baseline: +0.5% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ 100-rate-metrics-100-timesTime: ✅ 2.190ms (SLO: <2.550ms 📉 -14.1%) vs baseline: +0.4% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ flush-1-metricTime: ✅ 4.605µs (SLO: <20.000µs 📉 -77.0%) vs baseline: +0.5% Memory: ✅ 35.173MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.6% ✅ flush-100-metricsTime: ✅ 171.930µs (SLO: <250.000µs 📉 -31.2%) vs baseline: -1.7% Memory: ✅ 35.350MB (SLO: <35.500MB 🟡 -0.4%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.170ms (SLO: <2.500ms 📉 -13.2%) vs baseline: -1.2% Memory: ✅ 36.097MB (SLO: <36.500MB 🟡 -1.1%) vs baseline: +4.9% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
|
Closing this PR in favor of @KowalskiThomas's https://github.com/DataDog/dd-trace-py/pull/15762/files |
Description
Currently can't merge backports due to this error: #15759 (comment)
https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1314352152
From talking to @robertomonteromiguel, dd-trace-py needs to use the updated commit sha
However it looks like with the bumped up version, the error has changed to https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1314669337
Testing
Risks
Additional Notes
The error is still occurring with this fix attempt so there's another root cause.