-
Notifications
You must be signed in to change notification settings - Fork 475
feat(ci_visibility): release the new pytest plugin #15768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
Performance SLOsComparing candidate vitor-de-araujo/release-new-plugin (1cfdb92) with baseline main (240abfb) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.001µs (SLO: <20.000µs -10.0%) vs baseline: 📈 +20.9% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.7% ✅ add_inplace_aspectTime: ✅ 14.962µs (SLO: <20.000µs 📉 -25.2%) vs baseline: +0.9% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.6% ✅ add_inplace_noaspectTime: ✅ 0.340µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +1.2% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ add_noaspectTime: ✅ 0.547µs (SLO: <10.000µs 📉 -94.5%) vs baseline: ~same Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.6% ✅ bytearray_aspectTime: ✅ 17.958µs (SLO: <30.000µs 📉 -40.1%) vs baseline: -0.1% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ bytearray_extend_aspectTime: ✅ 23.967µs (SLO: <30.000µs 📉 -20.1%) vs baseline: +0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ bytearray_extend_noaspectTime: ✅ 2.739µs (SLO: <10.000µs 📉 -72.6%) vs baseline: -0.3% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ bytearray_noaspectTime: ✅ 1.492µs (SLO: <10.000µs 📉 -85.1%) vs baseline: +0.6% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ bytes_aspectTime: ✅ 16.765µs (SLO: <20.000µs 📉 -16.2%) vs baseline: +0.9% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ bytes_noaspectTime: ✅ 1.411µs (SLO: <10.000µs 📉 -85.9%) vs baseline: -1.7% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ bytesio_aspectTime: ✅ 55.511µs (SLO: <70.000µs 📉 -20.7%) vs baseline: +0.3% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ bytesio_noaspectTime: ✅ 3.269µs (SLO: <10.000µs 📉 -67.3%) vs baseline: +0.1% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 14.691µs (SLO: <20.000µs 📉 -26.5%) vs baseline: +0.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ capitalize_noaspectTime: ✅ 2.570µs (SLO: <10.000µs 📉 -74.3%) vs baseline: -0.1% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ casefold_aspectTime: ✅ 14.557µs (SLO: <20.000µs 📉 -27.2%) vs baseline: -1.0% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ casefold_noaspectTime: ✅ 3.164µs (SLO: <10.000µs 📉 -68.4%) vs baseline: -0.7% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ decode_aspectTime: ✅ 15.547µs (SLO: <30.000µs 📉 -48.2%) vs baseline: -0.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ decode_noaspectTime: ✅ 1.604µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.9% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 18.077µs (SLO: <30.000µs 📉 -39.7%) vs baseline: 📈 +21.8% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ encode_noaspectTime: ✅ 1.480µs (SLO: <10.000µs 📉 -85.2%) vs baseline: -2.1% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ format_aspectTime: ✅ 171.352µs (SLO: <200.000µs 📉 -14.3%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.7% ✅ format_map_aspectTime: ✅ 191.101µs (SLO: <200.000µs -4.4%) vs baseline: -0.1% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ format_map_noaspectTime: ✅ 3.824µs (SLO: <10.000µs 📉 -61.8%) vs baseline: +1.1% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.9% ✅ format_noaspectTime: ✅ 3.177µs (SLO: <10.000µs 📉 -68.2%) vs baseline: +0.3% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ index_aspectTime: ✅ 15.339µs (SLO: <20.000µs 📉 -23.3%) vs baseline: +0.4% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ index_noaspectTime: ✅ 0.465µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ join_aspectTime: ✅ 17.116µs (SLO: <20.000µs 📉 -14.4%) vs baseline: +0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.6% ✅ join_noaspectTime: ✅ 1.550µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -0.4% Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.7% ✅ ljust_aspectTime: ✅ 20.786µs (SLO: <30.000µs 📉 -30.7%) vs baseline: -0.2% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.7% ✅ ljust_noaspectTime: ✅ 2.708µs (SLO: <10.000µs 📉 -72.9%) vs baseline: ~same Memory: ✅ 42.546MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 17.802µs (SLO: <30.000µs 📉 -40.7%) vs baseline: -0.7% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ lower_noaspectTime: ✅ 2.426µs (SLO: <10.000µs 📉 -75.7%) vs baseline: -0.6% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ lstrip_aspectTime: ✅ 17.701µs (SLO: <30.000µs 📉 -41.0%) vs baseline: +0.4% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.9% ✅ lstrip_noaspectTime: ✅ 1.870µs (SLO: <10.000µs 📉 -81.3%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ modulo_aspectTime: ✅ 166.841µs (SLO: <200.000µs 📉 -16.6%) vs baseline: ~same Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.3% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.731µs (SLO: <200.000µs 📉 -12.6%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.5% ✅ modulo_aspect_for_bytesTime: ✅ 168.959µs (SLO: <200.000µs 📉 -15.5%) vs baseline: +0.3% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +5.2% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.112µs (SLO: <200.000µs 📉 -13.9%) vs baseline: +0.1% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ modulo_noaspectTime: ✅ 3.665µs (SLO: <10.000µs 📉 -63.3%) vs baseline: -0.7% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ replace_aspectTime: ✅ 212.207µs (SLO: <300.000µs 📉 -29.3%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <44.000MB -3.2%) vs baseline: +4.4% ✅ replace_noaspectTime: ✅ 2.907µs (SLO: <10.000µs 📉 -70.9%) vs baseline: +0.3% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ repr_aspectTime: ✅ 1.478µs (SLO: <10.000µs 📉 -85.2%) vs baseline: +4.3% Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.8% ✅ repr_noaspectTime: ✅ 0.523µs (SLO: <10.000µs 📉 -94.8%) vs baseline: -0.9% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ rstrip_aspectTime: ✅ 22.904µs (SLO: <30.000µs 📉 -23.7%) vs baseline: 📈 +20.4% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.5% ✅ rstrip_noaspectTime: ✅ 1.935µs (SLO: <10.000µs 📉 -80.6%) vs baseline: -0.5% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ slice_aspectTime: ✅ 15.940µs (SLO: <20.000µs 📉 -20.3%) vs baseline: +0.1% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ slice_noaspectTime: ✅ 0.597µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ stringio_aspectTime: ✅ 54.135µs (SLO: <80.000µs 📉 -32.3%) vs baseline: ~same Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 3.636µs (SLO: <10.000µs 📉 -63.6%) vs baseline: +0.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ strip_aspectTime: ✅ 17.688µs (SLO: <20.000µs 📉 -11.6%) vs baseline: +0.2% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 1.874µs (SLO: <10.000µs 📉 -81.3%) vs baseline: -0.2% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ swapcase_aspectTime: ✅ 18.500µs (SLO: <30.000µs 📉 -38.3%) vs baseline: +0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ swapcase_noaspectTime: ✅ 2.814µs (SLO: <10.000µs 📉 -71.9%) vs baseline: +0.8% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ title_aspectTime: ✅ 18.236µs (SLO: <30.000µs 📉 -39.2%) vs baseline: -0.5% Memory: ✅ 42.664MB (SLO: <43.000MB 🟡 -0.8%) vs baseline: +5.0% ✅ title_noaspectTime: ✅ 2.670µs (SLO: <10.000µs 📉 -73.3%) vs baseline: -0.4% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ translate_aspectTime: ✅ 20.582µs (SLO: <30.000µs 📉 -31.4%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ translate_noaspectTime: ✅ 4.333µs (SLO: <10.000µs 📉 -56.7%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 17.993µs (SLO: <30.000µs 📉 -40.0%) vs baseline: -0.3% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ upper_noaspectTime: ✅ 2.444µs (SLO: <10.000µs 📉 -75.6%) vs baseline: +0.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.193µs (SLO: <10.000µs 📉 -48.1%) vs baseline: 📈 +22.3% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ ospathbasename_noaspectTime: ✅ 4.308µs (SLO: <10.000µs 📉 -56.9%) vs baseline: +0.5% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 6.228µs (SLO: <10.000µs 📉 -37.7%) vs baseline: -0.5% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +5.0% ✅ ospathjoin_noaspectTime: ✅ 6.317µs (SLO: <10.000µs 📉 -36.8%) vs baseline: ~same Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 3.558µs (SLO: <10.000µs 📉 -64.4%) vs baseline: -0.3% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.7% ✅ ospathnormcase_noaspectTime: ✅ 3.654µs (SLO: <10.000µs 📉 -63.5%) vs baseline: -0.4% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ ospathsplit_aspectTime: ✅ 4.982µs (SLO: <10.000µs 📉 -50.2%) vs baseline: +1.4% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 5.010µs (SLO: <10.000µs 📉 -49.9%) vs baseline: -0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.744µs (SLO: <10.000µs 📉 -62.6%) vs baseline: -0.9% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.745µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.8% Memory: ✅ 42.487MB (SLO: <43.500MB -2.3%) vs baseline: +4.5% ✅ ospathsplitext_aspectTime: ✅ 4.638µs (SLO: <10.000µs 📉 -53.6%) vs baseline: +0.6% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 4.631µs (SLO: <10.000µs 📉 -53.7%) vs baseline: ~same Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.8% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.395µs (SLO: <20.000µs 📉 -83.0%) vs baseline: 📈 +13.1% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-count-metrics-100-timesTime: ✅ 201.086µs (SLO: <220.000µs -8.6%) vs baseline: +0.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.2% ✅ 1-distribution-metric-1-timesTime: ✅ 3.324µs (SLO: <20.000µs 📉 -83.4%) vs baseline: -1.4% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ 1-distribution-metrics-100-timesTime: ✅ 213.539µs (SLO: <230.000µs -7.2%) vs baseline: -0.8% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ 1-gauge-metric-1-timesTime: ✅ 2.213µs (SLO: <20.000µs 📉 -88.9%) vs baseline: ~same Memory: ✅ 34.977MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +5.1% ✅ 1-gauge-metrics-100-timesTime: ✅ 135.983µs (SLO: <150.000µs -9.3%) vs baseline: -0.5% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.6% ✅ 1-rate-metric-1-timesTime: ✅ 3.130µs (SLO: <20.000µs 📉 -84.4%) vs baseline: -0.6% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ 1-rate-metrics-100-timesTime: ✅ 214.521µs (SLO: <250.000µs 📉 -14.2%) vs baseline: ~same Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 19.828ms (SLO: <22.000ms -9.9%) vs baseline: -1.0% Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.233ms (SLO: <2.550ms 📉 -12.4%) vs baseline: +0.3% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.423ms (SLO: <1.550ms -8.2%) vs baseline: +1.7% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 100-rate-metrics-100-timesTime: ✅ 2.186ms (SLO: <2.550ms 📉 -14.3%) vs baseline: +0.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.4% ✅ flush-1-metricTime: ✅ 4.580µs (SLO: <20.000µs 📉 -77.1%) vs baseline: +1.2% Memory: ✅ 35.134MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 174.088µs (SLO: <250.000µs 📉 -30.4%) vs baseline: -0.2% Memory: ✅ 35.193MB (SLO: <35.500MB 🟡 -0.9%) vs baseline: +4.7% ✅ flush-1000-metricsTime: ✅ 2.177ms (SLO: <2.500ms 📉 -12.9%) vs baseline: -0.1% Memory: ✅ 36.097MB (SLO: <36.500MB 🟡 -1.1%) vs baseline: +4.9% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
…raujo/release-new-plugin
…og/dd-trace-py into vitor-de-araujo/release-new-plugin
Codeowners resolved as |
Description
This PR adds a release note about the new pytest plugin and documentation on how to enable it. The new plugin is disabled by default for now.
This PR also introduces a couple of changes:
_DD_CIVISIBILITY_USE_CI_CONTEXT_PROVIDERis used, we disable the integration with APM spans. This is needed in ddtrace's own CI pipeline for tests that check APM span generation._CI_DD_GIT_REPOSITORY_URLenv var that can be used in ddtrace's own CI to test the new plugin while sending the data as if it were different repo; this allows us to compare the output of the two plugins without mixing them.tests/tracer/test_logger.pyto use a module name different fromddtrace.testingas a test example, as this clashes with the new plugin's module name.time.time_ns()with theTimeclass, to avoid interference fromfreezegun.Testing
The new plugin was tested in PR #15742. This PR is basically the same code as PR #15742, without the
riotfile.pychanges that enable the new plugin in the ddtrace CI.Risks
None, as the new plugin is currently disabled by default.
Additional Notes
In a subsequent PR, we can apply the
riotfile.pychanges in PR #15742 to enable the new plugin in ddtrace's CI.