[docs/component-stability.md] Add criteria for graduating between stability levels #11864
Conversation
mx-psi
requested review from
atoulme,
codeboten,
bogdandrutu,
dmitryax and
TylerHelmuth
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #11864 +/- ##
=======================================
Coverage 91.44% 91.44%
=======================================
Files 487 487
Lines 26848 26848
=======================================
Hits 24551 24551
Misses 1814 1814
Partials 483 483 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Christos Markou <[email protected]>
| ## Beta to stable | ||
|
|
||
| To graduate any signal from beta to stable on a component: | ||
| 1. The component MUST have at least three active code owners. |
There was a problem hiding this comment.
I think there should be a commitment from codeowners that there is a SLA for first response on bug issues.
The commitment should be measured in days.
There was a problem hiding this comment.
This raises some questions for me including:
- What happens when people go on vacation/have a kid/[insert activity here that leads to a prolonged period of absence]?
- What happens if people don't follow this SLA? Typically an SLA means that you pay if you don't meet a certain standard, how do you "pay" here?
There was a problem hiding this comment.
It also raises the question of how we measure that. Do we have any automation in place for it?
There was a problem hiding this comment.
I think we shouldn't make it too hard to have community components.
I think vendor components and widely used components will not have any issue to follow the guidelines.
When we think about components that add value to the overall project, but may not be interesting/priority to vendors, they may struggle to get folks involved on it, and that would disqualify them from being moved to stable.
I do understand that we need to provide a way to ensure maintainability of stable components, but maybe we could draw something over the ideas of:
- Being active
- Replying to issues related to its components timely
- Fixing reported bug timely
- ...
If we have a component that is not vendor related, but maintained by 2 folks from a single employer, that wouldn't allow them to move on.
Also, let's imagine the following scenario:
- 3 folks are codeowners of a component, 2 from one company and another one from another company.
- They graduate to stable.
- A couple of months later the person that was from the other company moves to the same company of the other 2. Would the component be demoted?
I don't think it should be, if they are active and responsive in issues related to that component.
I know it is a corner case, and may never happen, but it still can.
My main point here, is actually that we shouldn't make it too hard to have community components.
I know a couple of companies that develop internal components to solve their customers' issues. It would be awesome to have a couple of those contributed back to upstream, and let the community grow together.
There is a trade-off between having more components and having fewer components that are more actively maintained. We need to be mindful of where we draw the line, but my feeling is that right now we have too many components that are not well maintained.
There's two questions to consider here:
On this PR I am focusing on (1). For doing (1), we need to focus on things that we can measure/check at the time of marking as stable. Some of the things you mention are, I feel like, important criteria for deciding if a component should be moved to unmaintained, but not to move to stable.
The point with the 'vendor diversity' is that I think it is a good predictor of component quality (more than one vendor means more focus on a wide number of use cases) and maintainability (we don't depend on a single company). Maybe it would help to do an analysis of existing components to see how difficult this is to achieve?
This PR makes no claims about when a component should be 'demoted'. Currently the only way to be demoted is to be moved to unmaintained, we have some rules about when that can happen. I personally don't think we should move from stable->beta, I think that would be confusing for end users. The way I see this it is a bit like the CNCF project status: there is no moving from graduated to incubating, only from graduated to deprecated. |
|
I split off part of this PR in #11937. PTAL at that one first |
Agree.
Makes sense.
Let's take the Connector first, from all components none of them have 3 ACTIVE codeowners.
On the other hand, I think this "rule" would bring more awareness to the components, and maybe it would also bring more codeowners to "important" components. Still not sure about the the amount of codeowners. |
After thinking further and discussing with @mx-psi, I believe this criteria is going to be the foundation for users to further engage and assume codeowner's responsibility in the components they would like to move to stable. |
…levels' section (#11937) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description Split off from #11864, describes how the graduation would work without any additional criteria. Rendered diagram: ```mermaid stateDiagram-v2 state Maintained { InDevelopment --> Alpha Alpha --> Beta Beta --> Stable } InDevelopment: In Development Maintained --> Unmaintained Unmaintained --> Maintained Maintained --> Deprecated Deprecated --> Maintained: (should be rare) ``` --------- Co-authored-by: Christos Markou <[email protected]>
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
#### Description I would like to become a codeowner of the awsfirehosereceiver. Elastic (where I work) intends to use this receiver heavily, and I would like to: - help spread out the maintenance load - ensure we have a path to progressing beyond the current Alpha stability Assuming open-telemetry/opentelemetry-collector#11864 goes through, it will be necessary to have multiple codeowners to progress to Beta and preferable to have codeowners from multiple employers, which is the case for @Aneurysm9 (AWS) and me (Elastic). Relevant changes: - #37111 (superseded by #37361) - #37262 - #37361 - #38388 (will extract from awsfirehosereceiver) - #38445 #### Link to tracking issue N/A #### Testing N/A #### Documentation N/A
There was a problem hiding this comment.
The criteria look reasonable; my only concern is whether our current automation can support approvers when reviewing Graduation requests.
For the most used components, 60 days is enough time for dozens of PRs/Issues to be opened, and code owners may have changed during this period. Asking approvers to look at all of them and consider code owner changes might be too difficult if done manually. If a process is too difficult to follow, people tend to bypass it 😅
| ## Beta to stable | ||
|
|
||
| To graduate any signal from beta to stable on a component: | ||
| 1. The component MUST have at least three active code owners. |
There was a problem hiding this comment.
It also raises the question of how we measure that. Do we have any automation in place for it?
|
@ArthurSens I think those concerns are reasonable, we definitely need more automation to be able to do this at scale and over time. I think the focus at this stage should be whether these requirements are automatable, we will automate things once we have the rules defined |
|
I intend to merge this in a couple of days if there are no further comments |
|
Are there any plans regarding "technical stability/maturity"? For instance, a threshold of test coverage and/or linting issues. I'm thinking here from a "user" point of view. What does stability mean to users? Number and activity of maintainers is a point, but I also believe there are some technical aspects to it that could be considered (or not). |
@douglascamata Yes! see #11553 for the full list of things I want to cover in this document. If you think the list is incomplete or you would like to help me put this into wording feel free to comment on the issues/file a PR :) |
|
@djaglowski I will wait a bit more in case you have further comments and merge this on Monday morning |
Description
Code ownership and maintenance of components continues to be an issue, with varying levels of support across contrib. As we approach 1.0 and the ability to mark components as stable, we want to make sure that components that we deem as 'stable' have a healthy community around them. We have three datapoints that we can leverage here: how many codeowners a component has, how diverse these are in terms of employers and how actively the codeowners have been responding to issues/PRs in the recent past.
We need criteria that
Some notes:
Link to tracking issue
Fixes #11850