-
Notifications
You must be signed in to change notification settings - Fork 475
ci(appsec): fix subprocess flakyness [backport 4.1] #15728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
## Problem FLAKY TESTS IDS: DD_0CHY3W DD_NMRUZV DD_UPYTSO DD_9ICU9D DD_084DFM The `test_subprocess.py` tests were flaky, failing intermittently with specific random seeds (e.g., `--randomly-seed=1219310116`). The failures manifested in two ways: 1. **`Pin.get_from(os)` returning `None`**: Tests failed with `AttributeError: 'NoneType' object has no attribute '_clone'` 2. **Incorrect span counts**: Tests expecting 1 span received 2 spans, or vice versa The root cause was **inadequate cleanup between tests**, leading to state pollution that caused subsequent tests to fail depending on execution order. ## Root Cause Analysis Through systematic investigation with the problematic random seed, we identified **six distinct issues**: ### 1. Missing `os.fork` in `unpatch()` ❌ - `os.fork` was patched but never unpatched - Left wrapped functions persisting between tests - **Impact**: Stale wrappers with outdated closure state ### 2. Missing Pin Removal in `unpatch()` ❌ - Pin objects were never removed from `os` and `subprocess` modules - **Impact**: Tests expecting no Pin would find leftover Pins from previous tests ### 3. Test Couldn't Handle None Pin ❌ - When `asm_config._load_modules=False`, `patch()` returns early without setting a Pin - Tests blindly called `Pin.get_from(os)._clone()` causing AttributeError - **Impact**: Immediate test failure ### 4. `test_unpatch` Tried to Use Removed Pin ❌ - After proper unpatch cleanup, Pin is removed - Test tried to get and clone a non-existent Pin - **Impact**: Test failure after fixing issue #2 ### 5. Incomplete Fixture Cleanup ❌ - `auto_unpatch` fixture only cleaned up **after** tests - State from previous tests polluted subsequent tests - **Impact**: Test failures dependent on execution order ## Testing Before Fix ``` $ scripts/run-tests --venv 15fbf61 -- -- tests/contrib/subprocess/test_subprocess.py --randomly-seed=1219310116 # Multiple failures: # - test_ossystem_disabled[config1][py3.10]: AttributeError: 'NoneType' object has no attribute '_clone' # - test_ossystem_disabled[config1][py3.10]: AssertionError: assert 2 == 1 (unexpected spans) # - test_unpatch[py3.11]: AttributeError: 'NoneType' object has no attribute '_clone' ``` After Fix ``` $ scripts/run-tests --venv 15fbf61 --venv 7ed64b0 -- -- tests/contrib/subprocess/test_subprocess.py --randomly-seed=1219310116 ``` ✅ Python 3.10: 488/488 tests passed ✅ Python 3.11: 485/485 tests passed Verification - ✅ Tests pass with the problematic random seed on both Python 3.10 and 3.11 - ✅ Tests pass in isolation - ✅ Tests pass without random seed - ✅ No regression in other test suites --- Related Issues Fixes flaky tests reported with random seed --randomly-seed=1219310116: - test_ossystem_disabled[config1][py3.10] - Pin.get_from returns None - test_ossystem_disabled[config1][py3.10] - Getting 2 spans instead of 1 - test_unpatch[py3.11] - 'NoneType' object has no attribute '_clone' (cherry picked from commit fa5ce13) Signed-off-by: Alberto Vara <[email protected]>
|
|
c3f307b to
b0a2a01
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 245 ± 2 ms. The average import time from base is: 248 ± 2 ms. The import time difference between this PR and base is: -2.31 ± 0.09 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate backport-15724-to-4.1 (2b19cfb) with baseline 4.1 (ca995a6) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 17.747µs (SLO: <20.000µs 📉 -11.3%) vs baseline: 📈 +22.7% Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.6% ✅ add_inplace_aspectTime: ✅ 14.435µs (SLO: <20.000µs 📉 -27.8%) vs baseline: -0.5% Memory: ✅ 43.037MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +5.0% ✅ add_inplace_noaspectTime: ✅ 0.337µs (SLO: <10.000µs 📉 -96.6%) vs baseline: ~same Memory: ✅ 43.096MB (SLO: <43.000MB +0.2%) vs baseline: +5.1% ✅ add_noaspectTime: ✅ 0.543µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -0.1% Memory: ✅ 42.802MB (SLO: <43.000MB 🟡 -0.5%) vs baseline: +4.2% ✅ bytearray_aspectTime: ✅ 17.282µs (SLO: <30.000µs 📉 -42.4%) vs baseline: +0.3% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 22.711µs (SLO: <30.000µs 📉 -24.3%) vs baseline: ~same Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytearray_extend_noaspectTime: ✅ 2.621µs (SLO: <10.000µs 📉 -73.8%) vs baseline: +0.3% Memory: ✅ 42.998MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 1.423µs (SLO: <10.000µs 📉 -85.8%) vs baseline: ~same Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ bytes_aspectTime: ✅ 16.054µs (SLO: <20.000µs 📉 -19.7%) vs baseline: +0.4% Memory: ✅ 42.920MB (SLO: <43.000MB 🟡 -0.2%) vs baseline: +4.5% ✅ bytes_noaspectTime: ✅ 1.379µs (SLO: <10.000µs 📉 -86.2%) vs baseline: -0.9% Memory: ✅ 42.979MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.7% ✅ bytesio_aspectTime: ✅ 52.201µs (SLO: <70.000µs 📉 -25.4%) vs baseline: -0.8% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ bytesio_noaspectTime: ✅ 3.140µs (SLO: <10.000µs 📉 -68.6%) vs baseline: -0.5% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +3.7% ✅ capitalize_aspectTime: ✅ 14.182µs (SLO: <20.000µs 📉 -29.1%) vs baseline: +0.8% Memory: ✅ 42.880MB (SLO: <43.000MB 🟡 -0.3%) vs baseline: +4.5% ✅ capitalize_noaspectTime: ✅ 2.529µs (SLO: <10.000µs 📉 -74.7%) vs baseline: +0.3% Memory: ✅ 42.900MB (SLO: <43.000MB 🟡 -0.2%) vs baseline: +4.4% ✅ casefold_aspectTime: ✅ 14.028µs (SLO: <20.000µs 📉 -29.9%) vs baseline: -0.5% Memory: ✅ 43.096MB (SLO: <43.000MB +0.2%) vs baseline: +5.0% ✅ casefold_noaspectTime: ✅ 3.014µs (SLO: <10.000µs 📉 -69.9%) vs baseline: -0.4% Memory: ✅ 42.762MB (SLO: <43.000MB 🟡 -0.6%) vs baseline: +4.2% ✅ decode_aspectTime: ✅ 14.972µs (SLO: <30.000µs 📉 -50.1%) vs baseline: +0.7% Memory: ✅ 42.880MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.4% ✅ decode_noaspectTime: ✅ 1.547µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -1.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.7% ✅ encode_aspectTime: ✅ 17.819µs (SLO: <30.000µs 📉 -40.6%) vs baseline: 📈 +24.9% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ encode_noaspectTime: ✅ 1.451µs (SLO: <10.000µs 📉 -85.5%) vs baseline: -0.5% Memory: ✅ 42.998MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.9% ✅ format_aspectTime: ✅ 168.970µs (SLO: <200.000µs 📉 -15.5%) vs baseline: ~same Memory: ✅ 43.018MB (SLO: <43.250MB 🟡 -0.5%) vs baseline: +4.5% ✅ format_map_aspectTime: ✅ 188.853µs (SLO: <200.000µs -5.6%) vs baseline: -0.2% Memory: ✅ 43.175MB (SLO: <43.500MB 🟡 -0.7%) vs baseline: +5.1% ✅ format_map_noaspectTime: ✅ 3.691µs (SLO: <10.000µs 📉 -63.1%) vs baseline: -0.5% Memory: ✅ 42.939MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 3.096µs (SLO: <10.000µs 📉 -69.0%) vs baseline: +0.4% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.2% ✅ index_aspectTime: ✅ 14.641µs (SLO: <20.000µs 📉 -26.8%) vs baseline: -0.4% Memory: ✅ 42.920MB (SLO: <43.250MB 🟡 -0.8%) vs baseline: +4.3% ✅ index_noaspectTime: ✅ 0.465µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.4% Memory: ✅ 42.979MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.9% ✅ join_aspectTime: ✅ 16.263µs (SLO: <20.000µs 📉 -18.7%) vs baseline: -0.2% Memory: ✅ 43.018MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ join_noaspectTime: ✅ 1.476µs (SLO: <10.000µs 📉 -85.2%) vs baseline: -0.4% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.1% ✅ ljust_aspectTime: ✅ 19.957µs (SLO: <30.000µs 📉 -33.5%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.6% ✅ ljust_noaspectTime: ✅ 2.621µs (SLO: <10.000µs 📉 -73.8%) vs baseline: -0.5% Memory: ✅ 42.841MB (SLO: <43.250MB 🟡 -0.9%) vs baseline: +4.5% ✅ lower_aspectTime: ✅ 17.205µs (SLO: <30.000µs 📉 -42.7%) vs baseline: +0.2% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.1% ✅ lower_noaspectTime: ✅ 2.328µs (SLO: <10.000µs 📉 -76.7%) vs baseline: -0.7% Memory: ✅ 42.743MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +4.0% ✅ lstrip_aspectTime: ✅ 16.903µs (SLO: <20.000µs 📉 -15.5%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <43.250MB 🟡 -0.7%) vs baseline: +4.9% ✅ lstrip_noaspectTime: ✅ 1.825µs (SLO: <10.000µs 📉 -81.8%) vs baseline: +0.8% Memory: ✅ 42.939MB (SLO: <43.000MB 🟡 -0.1%) vs baseline: +4.9% ✅ modulo_aspectTime: ✅ 164.608µs (SLO: <200.000µs 📉 -17.7%) vs baseline: -0.1% Memory: ✅ 43.096MB (SLO: <43.500MB 🟡 -0.9%) vs baseline: +5.0% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 172.320µs (SLO: <200.000µs 📉 -13.8%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.7% ✅ modulo_aspect_for_bytesTime: ✅ 165.985µs (SLO: <200.000µs 📉 -17.0%) vs baseline: -0.2% Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +4.6% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 174.703µs (SLO: <200.000µs 📉 -12.6%) vs baseline: +3.2% Memory: ✅ 43.155MB (SLO: <43.500MB 🟡 -0.8%) vs baseline: +4.9% ✅ modulo_noaspectTime: ✅ 3.579µs (SLO: <10.000µs 📉 -64.2%) vs baseline: +0.8% Memory: ✅ 42.762MB (SLO: <43.000MB 🟡 -0.6%) vs baseline: +3.9% ✅ replace_aspectTime: ✅ 212.301µs (SLO: <300.000µs 📉 -29.2%) vs baseline: +0.4% Memory: ✅ 43.273MB (SLO: <44.000MB 🟡 -1.7%) vs baseline: +5.1% ✅ replace_noaspectTime: ✅ 5.077µs (SLO: <10.000µs 📉 -49.2%) vs baseline: -0.4% Memory: ✅ 42.861MB (SLO: <43.000MB 🟡 -0.3%) vs baseline: +4.5% ✅ repr_aspectTime: ✅ 1.373µs (SLO: <10.000µs 📉 -86.3%) vs baseline: -0.5% Memory: ✅ 42.880MB (SLO: <43.500MB 🟡 -1.4%) vs baseline: +4.6% ✅ repr_noaspectTime: ✅ 0.521µs (SLO: <10.000µs 📉 -94.8%) vs baseline: -2.3% Memory: ✅ 42.664MB (SLO: <43.000MB 🟡 -0.8%) vs baseline: +4.2% ✅ rstrip_aspectTime: ✅ 18.231µs (SLO: <30.000µs 📉 -39.2%) vs baseline: +0.1% Memory: ✅ 42.880MB (SLO: <43.000MB 🟡 -0.3%) vs baseline: +4.7% ✅ rstrip_noaspectTime: ✅ 1.862µs (SLO: <10.000µs 📉 -81.4%) vs baseline: -0.3% Memory: ✅ 42.880MB (SLO: <43.000MB 🟡 -0.3%) vs baseline: +4.4% ✅ slice_aspectTime: ✅ 15.469µs (SLO: <20.000µs 📉 -22.7%) vs baseline: -0.4% Memory: ✅ 43.057MB (SLO: <43.000MB +0.1%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 0.594µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -0.1% Memory: ✅ 42.959MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.8% ✅ stringio_aspectTime: ✅ 51.162µs (SLO: <80.000µs 📉 -36.0%) vs baseline: -0.4% Memory: ✅ 43.077MB (SLO: <43.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 3.469µs (SLO: <10.000µs 📉 -65.3%) vs baseline: -0.8% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ strip_aspectTime: ✅ 16.817µs (SLO: <20.000µs 📉 -15.9%) vs baseline: -0.8% Memory: ✅ 42.998MB (SLO: <43.000MB 🟡 ~same) vs baseline: +5.0% ✅ strip_noaspectTime: ✅ 2.000µs (SLO: <10.000µs 📉 -80.0%) vs baseline: 📈 +11.3% Memory: ✅ 42.743MB (SLO: <43.000MB 🟡 -0.6%) vs baseline: +3.9% ✅ swapcase_aspectTime: ✅ 17.707µs (SLO: <30.000µs 📉 -41.0%) vs baseline: ~same Memory: ✅ 42.998MB (SLO: <43.000MB 🟡 ~same) vs baseline: +4.9% ✅ swapcase_noaspectTime: ✅ 2.734µs (SLO: <10.000µs 📉 -72.7%) vs baseline: +0.3% Memory: ✅ 42.900MB (SLO: <43.000MB 🟡 -0.2%) vs baseline: +4.3% ✅ title_aspectTime: ✅ 17.508µs (SLO: <30.000µs 📉 -41.6%) vs baseline: -0.2% Memory: ✅ 42.900MB (SLO: <43.000MB 🟡 -0.2%) vs baseline: +4.4% ✅ title_noaspectTime: ✅ 2.733µs (SLO: <10.000µs 📉 -72.7%) vs baseline: +6.2% Memory: ✅ 42.821MB (SLO: <43.000MB 🟡 -0.4%) vs baseline: +4.4% ✅ translate_aspectTime: ✅ 19.471µs (SLO: <30.000µs 📉 -35.1%) vs baseline: -0.3% Memory: ✅ 43.037MB (SLO: <43.000MB ~same) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 4.131µs (SLO: <10.000µs 📉 -58.7%) vs baseline: -0.6% Memory: ✅ 43.057MB (SLO: <43.000MB +0.1%) vs baseline: +5.0% ✅ upper_aspectTime: ✅ 17.201µs (SLO: <30.000µs 📉 -42.7%) vs baseline: -0.6% Memory: ✅ 43.018MB (SLO: <43.000MB ~same) vs baseline: +4.8% ✅ upper_noaspectTime: ✅ 2.345µs (SLO: <10.000µs 📉 -76.6%) vs baseline: -0.7% Memory: ✅ 42.802MB (SLO: <43.000MB 🟡 -0.5%) vs baseline: +4.4% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.049µs (SLO: <10.000µs 📉 -49.5%) vs baseline: 📈 +25.5% Memory: ✅ 41.504MB (SLO: <43.000MB -3.5%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 4.120µs (SLO: <10.000µs 📉 -58.8%) vs baseline: +1.1% Memory: ✅ 41.504MB (SLO: <43.000MB -3.5%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 5.914µs (SLO: <10.000µs 📉 -40.9%) vs baseline: -0.2% Memory: ✅ 41.386MB (SLO: <43.000MB -3.8%) vs baseline: +4.8% ✅ ospathjoin_noaspectTime: ✅ 5.991µs (SLO: <10.000µs 📉 -40.1%) vs baseline: +0.3% Memory: ✅ 41.425MB (SLO: <43.000MB -3.7%) vs baseline: +4.8% ✅ ospathnormcase_aspectTime: ✅ 3.339µs (SLO: <10.000µs 📉 -66.6%) vs baseline: +1.8% Memory: ✅ 41.465MB (SLO: <43.000MB -3.6%) vs baseline: +4.9% ✅ ospathnormcase_noaspectTime: ✅ 3.393µs (SLO: <10.000µs 📉 -66.1%) vs baseline: +1.0% Memory: ✅ 41.425MB (SLO: <43.000MB -3.7%) vs baseline: +4.8% ✅ ospathsplit_aspectTime: ✅ 4.635µs (SLO: <10.000µs 📉 -53.6%) vs baseline: +0.7% Memory: ✅ 41.425MB (SLO: <43.000MB -3.7%) vs baseline: +4.8% ✅ ospathsplit_noaspectTime: ✅ 4.757µs (SLO: <10.000µs 📉 -52.4%) vs baseline: +1.1% Memory: ✅ 41.465MB (SLO: <43.000MB -3.6%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.595µs (SLO: <10.000µs 📉 -64.1%) vs baseline: -0.5% Memory: ✅ 41.425MB (SLO: <43.000MB -3.7%) vs baseline: +5.1% ✅ ospathsplitdrive_noaspectTime: ✅ 0.758µs (SLO: <10.000µs 📉 -92.4%) vs baseline: +1.8% Memory: ✅ 41.524MB (SLO: <43.000MB -3.4%) vs baseline: +5.1% ✅ ospathsplitext_aspectTime: ✅ 4.428µs (SLO: <10.000µs 📉 -55.7%) vs baseline: +0.4% Memory: ✅ 41.445MB (SLO: <43.000MB -3.6%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 4.513µs (SLO: <10.000µs 📉 -54.9%) vs baseline: +0.6% Memory: ✅ 41.445MB (SLO: <43.000MB -3.6%) vs baseline: +4.8% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.352µs (SLO: <20.000µs 📉 -83.2%) vs baseline: 📈 +11.3% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-count-metrics-100-timesTime: ✅ 199.763µs (SLO: <220.000µs -9.2%) vs baseline: -0.8% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.374µs (SLO: <20.000µs 📉 -83.1%) vs baseline: +1.0% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ 1-distribution-metrics-100-timesTime: ✅ 216.686µs (SLO: <230.000µs -5.8%) vs baseline: -0.7% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.6% ✅ 1-gauge-metric-1-timesTime: ✅ 2.177µs (SLO: <20.000µs 📉 -89.1%) vs baseline: +0.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.275µs (SLO: <150.000µs -9.1%) vs baseline: ~same Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ 1-rate-metric-1-timesTime: ✅ 3.167µs (SLO: <20.000µs 📉 -84.2%) vs baseline: +0.5% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ 1-rate-metrics-100-timesTime: ✅ 213.179µs (SLO: <250.000µs 📉 -14.7%) vs baseline: -0.3% Memory: ✅ 34.937MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ 100-count-metrics-100-timesTime: ✅ 20.079ms (SLO: <22.000ms -8.7%) vs baseline: -1.7% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.308ms (SLO: <2.550ms -9.5%) vs baseline: -0.8% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.4% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.414ms (SLO: <1.550ms -8.8%) vs baseline: +0.7% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 100-rate-metrics-100-timesTime: ✅ 2.204ms (SLO: <2.550ms 📉 -13.6%) vs baseline: -0.8% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.3% ✅ flush-1-metricTime: ✅ 4.637µs (SLO: <20.000µs 📉 -76.8%) vs baseline: -0.7% Memory: ✅ 35.095MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.7% ✅ flush-100-metricsTime: ✅ 173.887µs (SLO: <250.000µs 📉 -30.4%) vs baseline: -0.4% Memory: ✅ 35.291MB (SLO: <35.500MB 🟡 -0.6%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.179ms (SLO: <2.500ms 📉 -12.8%) vs baseline: -0.2% Memory: ✅ 36.038MB (SLO: <36.500MB 🟡 -1.3%) vs baseline: +5.2% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
This reverts commit 8dd9c03.
This reverts commit 8dd9c03.
Backport fa5ce13 from #15724 to 4.1.
Problem
FLAKY TESTS IDS: DD_0CHY3W DD_NMRUZV DD_UPYTSO DD_9ICU9D DD_084DFM
The
test_subprocess.pytests were flaky, failing intermittently with specific random seeds (e.g.,--randomly-seed=1219310116). The failures manifested in two ways:Pin.get_from(os)returningNone: Tests failed withAttributeError: 'NoneType' object has no attribute '_clone'The root cause was inadequate cleanup between tests, leading to state pollution that caused subsequent tests to fail depending on execution order.
Root Cause Analysis
Through systematic investigation with the problematic random seed, we identified six distinct issues:
1. Missing
os.forkinunpatch()❌os.forkwas patched but never unpatched2. Missing Pin Removal in
unpatch()❌osandsubprocessmodules3. Test Couldn't Handle None Pin ❌
asm_config._load_modules=False,patch()returns early without setting a PinPin.get_from(os)._clone()causing AttributeError4.
test_unpatchTried to Use Removed Pin ❌5. Incomplete Fixture Cleanup ❌
auto_unpatchfixture only cleaned up after testsTesting
Before Fix
After Fix
✅ Python 3.10: 488/488 tests passed
✅ Python 3.11: 485/485 tests passed
Verification
Related Issues
Fixes flaky tests reported with random seed --randomly-seed=1219310116: