Skip to content

Rerun all failed tests, not only marked as fragile #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 17, 2022

Conversation

ylobankov
Copy link
Contributor

test-run supports functionality to rerun failed tests in place, but
these tests have to be on so called fragile list. To add a test to the
fragile list we need to add a special configuration to the suite.ini
file of a test suite. Configuration example:

fragile = {
    "retries": 5,
    "tests": {
        "tarantoolctl.test.lua": {
            "issues": [ "gh-5059", "gh-5346" ]
        },
        "debug.test.lua": {
            "issues": [ "gh-5346" ]
        },
        ...
    }
}

Rerunning failed tests in place is quite convenient because it allows us
to avoid rerunning all tests again and thus save time.

But to make it work as expected we should keep the list of fragile tests
always up-to-date. Flaky tests may be introduced every day and keeping
the list of fragile tests always up-to-date becomes extremely difficult
to do.

So our solusion is quite simple: just rerun all failed tests.
By default, the number of retries for regular and fragile tests is 3.
But for fragile tests this number can be overriden in the suite.ini
file.

Closes #328

@ylobankov ylobankov requested review from artembo and Totktonada March 15, 2022 08:46
Copy link

@artembo artembo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but upvoting NickVolynkin question

test-run supports functionality to rerun failed tests in place, but
these tests have to be on so called fragile list. To add a test to the
fragile list we need to add a special configuration to the suite.ini
file of a test suite. Configuration example:

    fragile = {
        "retries": 5,
        "tests": {
            "tarantoolctl.test.lua": {
                "issues": [ "gh-5059", "gh-5346" ]
            },
            "debug.test.lua": {
                "issues": [ "gh-5346" ]
            },
            ...
        }
    }

Rerunning failed tests in place is quite convenient because it allows us
to avoid rerunning all tests again and thus save time.

But to make it work as expected we should keep the list of fragile tests
always up-to-date. Flaky tests may be introduced every day and keeping
the list of fragile tests always up-to-date becomes extremely difficult
to do.

So our solusion is quite simple: just rerun all failed tests.
By default, the number of retries for regular and fragile tests is 3.
But for fragile tests this number can be overriden in the suite.ini
file.

Closes #328
@ylobankov ylobankov force-pushed the ylobankov/retry-all-failed-tests branch from 5ea1791 to 36cee89 Compare March 16, 2022 20:36
Copy link
Member

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM visually.

@NickVolynkin NickVolynkin self-requested a review March 17, 2022 08:30
@kyukhin kyukhin merged commit 0915679 into master Mar 17, 2022
@kyukhin kyukhin deleted the ylobankov/retry-all-failed-tests branch March 17, 2022 08:52
kyukhin added a commit to kyukhin/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 18, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 21, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 21, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 21, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 21, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

(cherry picked from commit 0b46154)
kyukhin added a commit to tarantool/tarantool that referenced this pull request Mar 21, 2022
Bump test-run to new version with the following improvements:
- Add timeout for starting tarantool server (tarantool/test-run#302)
- Kill hanging processes of not started servers (tarantool/test-run#332)
- Rerun all failed tests, not only marked as fragile
  (tarantool/test-run#329)

NO_DOC=testing
NO_TEST=testing
NO_CHANGELOG=testing

(cherry picked from commit 0b46154)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Retry tests on failure
5 participants