Description
The short story
The CPython CI is growing quickly, we'd like to try out some monitoring tools.
Long version
The CPython GitHub Actions CI is getting more complex. We're testing more operating systems and architectures, and things like free-threaded, JIT, address and thread sanitisation, fuzzing, docs. For example:
-
On the one hand, automated testing is good, it prevents bugs and regressions in a fast-moving project, and means we can better support the hardware used by our users (e.g. aarch64-apple-darwin was just added to tier 1 in PEP 11).
-
On the other, the CI is getting more complex and taking longer to run. We're on the GitHub Free plan which limits us to 5 concurrent macOS jobs. During busy periods this has caused builds to wait ~100 minutes before starting, and to mitigate we've begun reducing some testing (CI: Only test free-threading with faster macOS M1 cpython#116814).
Monitoring could help us see where we're using CI resources and help manage it. Hopefully it would help us see how long different jobs take, how long workflows take within jobs, and so on.
Datadog have a tool called CI Visibility: https://www.datadoghq.com/blog/datadog-github-actions-ci-visibility/ I've not tried it before, but it looks useful.
Datadog also have an Open Source Partner Program to offer free accounts, and I see the PSF is already a partner, using it to monitor PyPI: https://opensource.datadoghq.com/
@ewdurbin Is CI Visibility available for free under the current PSF account and would we be able to try it out for https://github.com/python/cpython? (Config docs)
If not, what are the next steps to set up a demo? Thank you!