-
Notifications
You must be signed in to change notification settings - Fork 17
Rerun all failed tests, not only marked as fragile #329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
artembo
approved these changes
Mar 15, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, but upvoting NickVolynkin question
test-run supports functionality to rerun failed tests in place, but these tests have to be on so called fragile list. To add a test to the fragile list we need to add a special configuration to the suite.ini file of a test suite. Configuration example: fragile = { "retries": 5, "tests": { "tarantoolctl.test.lua": { "issues": [ "gh-5059", "gh-5346" ] }, "debug.test.lua": { "issues": [ "gh-5346" ] }, ... } } Rerunning failed tests in place is quite convenient because it allows us to avoid rerunning all tests again and thus save time. But to make it work as expected we should keep the list of fragile tests always up-to-date. Flaky tests may be introduced every day and keeping the list of fragile tests always up-to-date becomes extremely difficult to do. So our solusion is quite simple: just rerun all failed tests. By default, the number of retries for regular and fragile tests is 3. But for fragile tests this number can be overriden in the suite.ini file. Closes #328
5ea1791
to
36cee89
Compare
Totktonada
reviewed
Mar 16, 2022
Totktonada
approved these changes
Mar 16, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM visually.
NickVolynkin
approved these changes
Mar 17, 2022
kyukhin
added a commit
to kyukhin/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 18, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 21, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 21, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 21, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 21, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing (cherry picked from commit 0b46154)
kyukhin
added a commit
to tarantool/tarantool
that referenced
this pull request
Mar 21, 2022
Bump test-run to new version with the following improvements: - Add timeout for starting tarantool server (tarantool/test-run#302) - Kill hanging processes of not started servers (tarantool/test-run#332) - Rerun all failed tests, not only marked as fragile (tarantool/test-run#329) NO_DOC=testing NO_TEST=testing NO_CHANGELOG=testing (cherry picked from commit 0b46154)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
test-run supports functionality to rerun failed tests in place, but
these tests have to be on so called fragile list. To add a test to the
fragile list we need to add a special configuration to the suite.ini
file of a test suite. Configuration example:
Rerunning failed tests in place is quite convenient because it allows us
to avoid rerunning all tests again and thus save time.
But to make it work as expected we should keep the list of fragile tests
always up-to-date. Flaky tests may be introduced every day and keeping
the list of fragile tests always up-to-date becomes extremely difficult
to do.
So our solusion is quite simple: just rerun all failed tests.
By default, the number of retries for regular and fragile tests is 3.
But for fragile tests this number can be overriden in the suite.ini
file.
Closes #328