fix logFailedTestCases for RerunFormatter #2292
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jenisys
reviewed
Jun 17, 2023
michael-lloyd-morris
approved these changes
Jun 19, 2023
Contributor
michael-lloyd-morris
left a comment
michael-lloyd-morris
left a comment
There was a problem hiding this comment.
Overall looks great. Some minor doc fixing in the changelog, but that could be fixed during merge I think.
davidjgoss
requested changes
Jun 20, 2023
Contributor
davidjgoss
left a comment
davidjgoss
left a comment
There was a problem hiding this comment.
Thanks for doing this, looks good! Just a few things I've noted that would be good to change. Happy to help if you need it.
Before doing my modifications, I wanted to add a test about the incorrect behavior discussed in cucumber#2290 I will update this test in a later patch that fixes the issue. Signed-off-by: Antal Nemes <[email protected]>
This change makes followup patch a bit more readable. Signed-off-by: Antal Nemes <[email protected]>
Prior this patch, RerunFormatter should log unstable tests: tests that failed at any retry attempt. Checking the `willBeRetried` value makes sure we will not log failures for rerun attempts that will be retried. Referring: cucumber#2290 for more information. Signed-off-by: Antal Nemes <[email protected]>
Introducing `getFailureMap()` and `formatFailedTestCases()` to decouple console logging with the calculation. This would help users who want to customize RerunFormatter For more information: cucumber#2290 Signed-off-by: Antal Nemes <[email protected]>
separator is not part of a complex logic, neither a complex structure. Users may want to reuse it for customizing RerunFormatter. For more details: cucumber#2290 Signed-off-by: Antal Nemes <[email protected]>
Signed-off-by: Antal Nemes <[email protected]>
davidjgoss
approved these changes
Jun 22, 2023
Contributor
|
Hi @furiel, Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾 In return for this generous offer we hope you will:
On behalf of the Cucumber core team, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Referring to #2290
What's changed?
RerunFormatterused to log failed attempts even if the attempt was retried. For example, it was possible that a testcase might have been fixed with one of the retry attempts, but still was logged due to earlier failing attempt. From now,RerunFormatterdoes not log attempts that are retried.extendsRerunFormatter.What's your motivation?
I am using
retryto stabilize tests of a distributed system. It was possible (for example right after a deployment), that the first couple of test attempts are failed against a service. I am also usingRerunFormatterto log the failures. However, asRerunFormatterlogged the failed attempts too, it was difficult to find out what testcases I needed to look at in case of a regression.I attempted to
extendRerunFormatteraccording to my needs, which turned out to be more complex than I anticipated.I created an issue and this patchet about both behavior of the
RerunFormatter, and the challenges of it's customization.What kind of change is this?
RerunFormatter'slogFailedTestCasesoutput is changed.RerunFormatterto simplifyextends-ing from it.Anything particular you want feedback on?
This is my first time I am working with typescript/javascript. I appreciate any feedbacks, let it be small or complex. However, I may need a bit more guidance due to my infamiliarity with the language. Do not hesitate to overexplain the change requests. :)
I also set
Allow edits by maintainersif you feel something is easier for you to change, instead of guiding me through.Checklist: