-
Notifications
You must be signed in to change notification settings - Fork 476
ci(iast): fix flaky benchmark test [backport 4.1] #15765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Fix flakyness in `lstrip_aspect` https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1310950874 (cherry picked from commit 2484164) Signed-off-by: Alberto Vara <[email protected]>
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 245 ± 4 ms. The average import time from base is: 248 ± 4 ms. The import time difference between this PR and base is: -3.1 ± 0.2 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate backport-15756-to-4.1 (2eba529) with baseline 4.1 (c815cbf) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 18.167µs (SLO: <20.000µs -9.2%) vs baseline: 📈 +21.2% Memory: ✅ 42.762MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.6% ✅ add_inplace_aspectTime: ✅ 15.065µs (SLO: <20.000µs 📉 -24.7%) vs baseline: +0.3% Memory: ✅ 42.743MB (SLO: <43.250MB 🟡 -1.2%) vs baseline: +4.4% ✅ add_inplace_noaspectTime: ✅ 0.338µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -0.7% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.3% ✅ add_noaspectTime: ✅ 0.547µs (SLO: <10.000µs 📉 -94.5%) vs baseline: -0.4% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.6% ✅ bytearray_aspectTime: ✅ 17.877µs (SLO: <30.000µs 📉 -40.4%) vs baseline: -0.8% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ bytearray_extend_aspectTime: ✅ 23.817µs (SLO: <30.000µs 📉 -20.6%) vs baseline: ~same Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.1% ✅ bytearray_extend_noaspectTime: ✅ 2.644µs (SLO: <10.000µs 📉 -73.6%) vs baseline: -0.5% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ bytearray_noaspectTime: ✅ 1.442µs (SLO: <10.000µs 📉 -85.6%) vs baseline: ~same Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.5% ✅ bytes_aspectTime: ✅ 16.757µs (SLO: <20.000µs 📉 -16.2%) vs baseline: +0.1% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.4% ✅ bytes_noaspectTime: ✅ 1.395µs (SLO: <10.000µs 📉 -86.0%) vs baseline: -1.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ bytesio_aspectTime: ✅ 55.169µs (SLO: <70.000µs 📉 -21.2%) vs baseline: -0.6% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.2% ✅ bytesio_noaspectTime: ✅ 3.215µs (SLO: <10.000µs 📉 -67.9%) vs baseline: +0.5% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ capitalize_aspectTime: ✅ 14.708µs (SLO: <20.000µs 📉 -26.5%) vs baseline: +0.5% Memory: ✅ 42.939MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +5.2% ✅ capitalize_noaspectTime: ✅ 2.546µs (SLO: <10.000µs 📉 -74.5%) vs baseline: -1.2% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.9% ✅ casefold_aspectTime: ✅ 14.694µs (SLO: <20.000µs 📉 -26.5%) vs baseline: +0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ casefold_noaspectTime: ✅ 3.088µs (SLO: <10.000µs 📉 -69.1%) vs baseline: +1.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ decode_aspectTime: ✅ 15.561µs (SLO: <30.000µs 📉 -48.1%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.6% ✅ decode_noaspectTime: ✅ 1.603µs (SLO: <10.000µs 📉 -84.0%) vs baseline: ~same Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ encode_aspectTime: ✅ 18.249µs (SLO: <30.000µs 📉 -39.2%) vs baseline: 📈 +22.1% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ encode_noaspectTime: ✅ 1.477µs (SLO: <10.000µs 📉 -85.2%) vs baseline: -0.8% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ format_aspectTime: ✅ 170.410µs (SLO: <200.000µs 📉 -14.8%) vs baseline: -0.4% Memory: ✅ 42.802MB (SLO: <43.250MB 🟡 -1.0%) vs baseline: +5.1% ✅ format_map_aspectTime: ✅ 191.015µs (SLO: <200.000µs -4.5%) vs baseline: -0.2% Memory: ✅ 42.979MB (SLO: <43.500MB 🟡 -1.2%) vs baseline: +4.6% ✅ format_map_noaspectTime: ✅ 3.718µs (SLO: <10.000µs 📉 -62.8%) vs baseline: -0.7% Memory: ✅ 42.566MB (SLO: <43.250MB 🟡 -1.6%) vs baseline: +4.5% ✅ format_noaspectTime: ✅ 3.178µs (SLO: <10.000µs 📉 -68.2%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.8% ✅ index_aspectTime: ✅ 15.327µs (SLO: <20.000µs 📉 -23.4%) vs baseline: -0.3% Memory: ✅ 42.703MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +4.4% ✅ index_noaspectTime: ✅ 0.465µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +1.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 16.935µs (SLO: <20.000µs 📉 -15.3%) vs baseline: -0.2% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ join_noaspectTime: ✅ 1.482µs (SLO: <10.000µs 📉 -85.2%) vs baseline: +0.2% Memory: ✅ 42.625MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +4.6% ✅ ljust_aspectTime: ✅ 20.780µs (SLO: <30.000µs 📉 -30.7%) vs baseline: -0.2% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +4.6% ✅ ljust_noaspectTime: ✅ 2.684µs (SLO: <10.000µs 📉 -73.2%) vs baseline: +0.9% Memory: ✅ 42.782MB (SLO: <43.250MB 🟡 -1.1%) vs baseline: +5.4% ✅ lower_aspectTime: ✅ 17.940µs (SLO: <30.000µs 📉 -40.2%) vs baseline: +0.4% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.1% ✅ lower_noaspectTime: ✅ 2.370µs (SLO: <10.000µs 📉 -76.3%) vs baseline: -0.8% Memory: ✅ 42.703MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.1% ✅ lstrip_aspectTime: ✅ 17.631µs (SLO: <30.000µs 📉 -41.2%) vs baseline: -0.6% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +4.6% ✅ lstrip_noaspectTime: ✅ 1.815µs (SLO: <10.000µs 📉 -81.8%) vs baseline: -1.3% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ modulo_aspectTime: ✅ 167.052µs (SLO: <200.000µs 📉 -16.5%) vs baseline: +0.5% Memory: ✅ 42.920MB (SLO: <43.500MB 🟡 -1.3%) vs baseline: +5.3% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 174.783µs (SLO: <200.000µs 📉 -12.6%) vs baseline: ~same Memory: ✅ 43.037MB (SLO: <43.500MB 🟡 -1.1%) vs baseline: +5.2% ✅ modulo_aspect_for_bytesTime: ✅ 168.304µs (SLO: <200.000µs 📉 -15.8%) vs baseline: +0.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 172.015µs (SLO: <200.000µs 📉 -14.0%) vs baseline: ~same Memory: ✅ 42.821MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +4.8% ✅ modulo_noaspectTime: ✅ 3.725µs (SLO: <10.000µs 📉 -62.8%) vs baseline: +1.4% Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ replace_aspectTime: ✅ 214.344µs (SLO: <300.000µs 📉 -28.6%) vs baseline: +0.2% Memory: ✅ 42.861MB (SLO: <44.000MB -2.6%) vs baseline: +4.3% ✅ replace_noaspectTime: ✅ 5.236µs (SLO: <10.000µs 📉 -47.6%) vs baseline: +0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ repr_aspectTime: ✅ 1.389µs (SLO: <10.000µs 📉 -86.1%) vs baseline: +0.8% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ repr_noaspectTime: ✅ 0.522µs (SLO: <10.000µs 📉 -94.8%) vs baseline: -0.6% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 23.008µs (SLO: <30.000µs 📉 -23.3%) vs baseline: 📈 +20.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ rstrip_noaspectTime: ✅ 1.898µs (SLO: <10.000µs 📉 -81.0%) vs baseline: +1.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ slice_aspectTime: ✅ 16.087µs (SLO: <20.000µs 📉 -19.6%) vs baseline: +0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.5% ✅ slice_noaspectTime: ✅ 0.598µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.4% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ stringio_aspectTime: ✅ 53.804µs (SLO: <80.000µs 📉 -32.7%) vs baseline: +0.4% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.2% ✅ stringio_noaspectTime: ✅ 3.542µs (SLO: <10.000µs 📉 -64.6%) vs baseline: +0.7% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ strip_aspectTime: ✅ 17.566µs (SLO: <20.000µs 📉 -12.2%) vs baseline: -0.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ strip_noaspectTime: ✅ 1.833µs (SLO: <10.000µs 📉 -81.7%) vs baseline: +0.2% Memory: ✅ 42.762MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +4.7% ✅ swapcase_aspectTime: ✅ 18.447µs (SLO: <30.000µs 📉 -38.5%) vs baseline: +0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.2% ✅ swapcase_noaspectTime: ✅ 2.757µs (SLO: <10.000µs 📉 -72.4%) vs baseline: -1.7% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 18.287µs (SLO: <30.000µs 📉 -39.0%) vs baseline: +0.3% Memory: ✅ 42.605MB (SLO: <43.000MB 🟡 -0.9%) vs baseline: +4.4% ✅ title_noaspectTime: ✅ 2.628µs (SLO: <10.000µs 📉 -73.7%) vs baseline: -0.9% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ translate_aspectTime: ✅ 20.363µs (SLO: <30.000µs 📉 -32.1%) vs baseline: -0.3% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +4.3% ✅ translate_noaspectTime: ✅ 4.244µs (SLO: <10.000µs 📉 -57.6%) vs baseline: +0.3% Memory: ✅ 42.841MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +5.6% ✅ upper_aspectTime: ✅ 17.911µs (SLO: <30.000µs 📉 -40.3%) vs baseline: -0.2% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.5% ✅ upper_noaspectTime: ✅ 2.365µs (SLO: <10.000µs 📉 -76.4%) vs baseline: -1.2% Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.182µs (SLO: <10.000µs 📉 -48.2%) vs baseline: 📈 +22.0% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathbasename_noaspectTime: ✅ 4.267µs (SLO: <10.000µs 📉 -57.3%) vs baseline: -0.8% Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathjoin_aspectTime: ✅ 6.143µs (SLO: <10.000µs 📉 -38.6%) vs baseline: ~same Memory: ✅ 41.445MB (SLO: <43.500MB -4.7%) vs baseline: +4.9% ✅ ospathjoin_noaspectTime: ✅ 6.182µs (SLO: <10.000µs 📉 -38.2%) vs baseline: -0.9% Memory: ✅ 41.347MB (SLO: <43.500MB -5.0%) vs baseline: +4.3% ✅ ospathnormcase_aspectTime: ✅ 3.511µs (SLO: <10.000µs 📉 -64.9%) vs baseline: +1.1% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.9% ✅ ospathnormcase_noaspectTime: ✅ 3.590µs (SLO: <10.000µs 📉 -64.1%) vs baseline: +0.3% Memory: ✅ 41.327MB (SLO: <43.500MB -5.0%) vs baseline: +4.5% ✅ ospathsplit_aspectTime: ✅ 4.852µs (SLO: <10.000µs 📉 -51.5%) vs baseline: +0.3% Memory: ✅ 41.406MB (SLO: <43.500MB -4.8%) vs baseline: +4.8% ✅ ospathsplit_noaspectTime: ✅ 4.938µs (SLO: <10.000µs 📉 -50.6%) vs baseline: -0.2% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.768µs (SLO: <10.000µs 📉 -62.3%) vs baseline: +0.2% Memory: ✅ 41.386MB (SLO: <43.500MB -4.9%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.751µs (SLO: <10.000µs 📉 -92.5%) vs baseline: -0.8% Memory: ✅ 41.366MB (SLO: <43.500MB -4.9%) vs baseline: +4.7% ✅ ospathsplitext_aspectTime: ✅ 4.635µs (SLO: <10.000µs 📉 -53.7%) vs baseline: +0.2% Memory: ✅ 41.465MB (SLO: <43.500MB -4.7%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 4.674µs (SLO: <10.000µs 📉 -53.3%) vs baseline: -0.3% Memory: ✅ 41.504MB (SLO: <43.500MB -4.6%) vs baseline: +5.3% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
589d541 to
ad4db3c
Compare
Backport 2484164 from #15756 to 4.1.
Fix flakyness in
lstrip_aspecthttps://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/1310950874