[SUREFIRE-2065] Test Reports Inconsistencies with Parameterized and junit4

[jira-importer]

Christian Marquez Grabia opened SUREFIRE-2065 and commented

Running parameterized tests with the junit-platform provider causes tests to be counted as single test (unlike using junit4 provider). This makes the tests to be marked as 'Flaky' and marks the build as successful if the retry failed count is greater than zero. Reports also show a single test run even if there were multiple tests with different parameters.

For a sample project with the problem see surefire-flaky-issue 

This is also a possibly related to issue SUREFIRE-2010 and may help in the resolution of it.

---

Affects: 3.0.0-M5, 3.0.0-M6, 3.0.0-M7

Issue Links:

• SUREFIRE-2087 rerunFailingTestsCount incorrectly marks failed parametrized test as pass
  ("is duplicated by")

Remote Links:

• GitHub Pull Request #516
  
• GitHub Pull Request #544
  
• GitHub Pull Request #608
  

1 votes, 4 watchers

[jira-importer]

Aaron Braunstein commented

This issue also causes @ParameterizedTest name to have no effect in junit 5 tests right?

public static Stream<String> params() {
  return Stream.of("Hello World");
}
@ParameterizedTest(name = "{index}: {0}")
@MethodSource("params")
public void test(String param) {} 

Expected: test(String)[1: Hello World]
Actual: test(String)[1]

[jira-importer]

Christian Marquez Grabia commented

When creating the reports it seems that the need is to have a unique way of identifying the test being executed. If the test executed does not have a unique identifier then other problems would come up (what caused the initial issue here). It would mistakenly consider multiple tests to be one leading to other problems.

I am not 100% sure when this 'displayName' should be used (other than in your IDE when running the tests) ... maybe the log should show with proper display name, but I am still seeing the unique name instead, so this may still be a problem even after this fix

[jira-importer]

Olivier Lamy commented

PR merged. Thanks