[SUREFIRE-2010] Parameterized Selection Does not Work

[jira-importer]

David Georg Reichelt opened SUREFIRE-2010 and commented

In the current version (and also M6-SNAPSHOT), maven surefire is not capable of selecting parameterized tests based on the index. In https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html, it is described that this should work by providing the index, e.g. using 

-Dtest=MyTest#method[$INDEX]

 or 

-Dtest=MyTest#method[*]

 for all.

 

This happens both for JUnit 4 and JUnit 5.

I created a mwe demonstrating this problem: https://github.com/DaGeRe/parameterized-selection-demo

As far as I see it, the TestMethodFilter in  https://github.com/apache/maven-surefire/blob/master/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/TestMethodFilter.java#L45 does the filtering, but has only a descriptor in the form:

 

[engine:junit-jupiter]/[class:de.dagere.peass.ExampleTest]/[test-template:test(int)]

 

So there is not the concrete value, but only the information that an int should be provided. Therefore, I currently see not any option to fix this easily or get this running using a regex pattern.

Do I oversee something, or is it planned to fix this? If not, it would be better to update the documentation site accordingly.

---

Affects: 3.0.0-M5

Remote Links:

• GitHub Pull Request #476
  

1 votes, 7 watchers

[jira-importer]

Tibor Digana commented

DaGeRe1989
Such pattern -Dtest=MyTest#method[*] is not supported by the Surefire JUnit5 Provider.
See the documentation. It talks about the old versions of JUnit4.

[jira-importer]

David Georg Reichelt commented

Thanks for the hint, so JUnit 5 is not supported currently.

However, as far as I see, this also does not work for JUnit 4. So in the example project, running

mvn clean test -Dtest=*ExampleTestJUnit4#test[1]

also results in an error.

Using JUnit 4, the descriptor has the form

[engine:junit-vintage]/[runner:de.dagere.peass.ExampleTestJUnit4]/[test:%5B1%5D]/[test:test%5B1%5D(de.dagere.peass.ExampleTestJUnit4)]

[jira-importer]

Tibor Digana commented

DaGeRe1989
The biggest problem is that we do not have any integration test for this scenario with JUnit4 and your command mvn test -Dtest=*ExampleTestJUnit4#test[1].
This feature was introduced together with re-run in the commit be9d923.
You can extend the existing IT with a new test method on Github and show us if it does not work. Alternatively, you can use the old commit and prove that it works and find the commit with a regression after that. This process would really help us. Thx

[jira-importer]

Tibor Digana commented

DaGeRe1989
It works for me, see this XML report.

  <testcase name="test[1]" classname="de.dagere.peass.ExampleTestJUnit4" time="0.011">
    <system-out><![CDATA[1
]]></system-out>
  </testcase>

and console output:

[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ parameterized-selection-demo ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running de.dagere.peass.ExampleTestJUnit4
1
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.118 s - in de.dagere.peass.ExampleTestJUnit4
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.788 s
[INFO] Finished at: 2022-02-14T10:01:25+01:00
[INFO] ------------------------------------------------------------------------

As I said before, the filter with parameterized test works with JUnit4 and it is written in the documentation. We do not support it in JUnit5 yet.

Every user should understand why it's that.
Once you declare JUnit5 dependency, the plugin would use surefire-junit-platform.
If you delete the Jupiter engine dependency and keep the JUnit 4.13.2, the provider surefire-junit4 is used and then these patterns are supported which is the meaning written in the documentation.

[jira-importer]

Tibor Digana commented

DaGeRe1989
Pls read my previous comment. You wrote a good repro steps on GH. Would you utilize that and write an IT based on your code in Surefire project?
Regarding the same feature in surefire-junit-platform provider, you should debug the provider and try to dig into the code. Possibly you would have chance to understand the root cause and provide a fix.

[jira-importer]

David Georg Reichelt commented

Thanks for the feedback.

I can reproduce this, so it also seems to me like the problem is including the JUnit 5 engine, regardless of whether it is a JUnit 4 or JUnit 5 test.

I'll try to write an IT, but unfortunately, I'm busy until Wednesday, so I'll catch up on this on Thursday.

[jira-importer]

David Georg Reichelt commented

Unfortunately, I did not succeed to write a working integration test.

I looked at https://maven.apache.org/surefire/maven-surefire-plugin/developing.html, which stated that I should to

cd surefire-integration-tests/src/test/resources/failsafe-buildfail
mvn -Dsurefire.version=2.12 verify 

This did not work (and the link provided there is also not working), since the folder now seems to be named

surefire-its 

Could you fix this, or should I just change https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/site/apt/developing.apt.vm and create a PR?

Doing

surefire-its/src/test/resources/surefire-1146-rerunFailingTests-with-Parameterized
mvn -Dsurefire.version=3.0.0-M6-SNAPSHOT verify

works fine for me and yields the output I would expect.

Now I would like to write the integration test itself. After some fiddeling, I got the old test running by

mvn verify -Dsurefire.version=3.0.0-M6-SNAPSHOT -Dtest=Surefire1146RerunFailedAndParameterizedIT

Unfortunately, for my new test https://github.com/DaGeRe/maven-surefire/blob/surefire-2010/surefire-its/src/test/java/org/apache/maven/surefire/its/jiras/Surefire2010ParameterizedSelectionDoesNotWorkIT.java (that currently does not select anything and should just execute the tests), the output of

mvn verify -Dsurefire.version=3.0.0-M6-SNAPSHOT  -Dtest=Surefire2010ParameterizedSelectionDoesNotWorkIT

is

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project jiras-surefire-2010: Compilation failure                  
[ERROR] /tmp/Surefire2010ParameterizedSelectionDoesNotWorkIT_testJUnit4/src/test/java/junitplatform/ParametersTest.java:[37,22] static interface method invocations are not supported in 
-source 7                                                                                                                                                                                
[ERROR]   (use -source 8 or higher to enable static interface method invocations)                                                                                                        
[ERROR] -> [Help 1]                                                                                                                                                                      
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project jiras-sure
fire-2010: Compilation failure                                                                                                                                                           
/tmp/Surefire2010ParameterizedSelectionDoesNotWorkIT_testJUnit4/src/test/java/junitplatform/ParametersTest.java:[37,22] static interface method invocations are not supported in -source 7 

I tried to fix this by adding

<properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

to the

surefire-its/src/test/resources/surefire-2010-parameterized-selection-does-not-work/pom.xml

but then I got the following output:

[ERROR] junitplatform.ParametersTest.testOneFailingPassingTest(ConnectionPoolFactory)[2]  Time elapsed: 0.005 s  <<< FAILURE!                                                            
org.opentest4j.AssertionFailedError: expected: <1> but was: <2>                                                                                                                          
        at junitplatform.ParametersTest.testOneFailingPassingTest(ParametersTest.java:57)                                                                                                
[INFO] Running junitplatform.FlakyFirstTimeTest                                                                                                                                          
Failing test                                                                                                                                                                             
Error test                                                                                                                                                                               
Passing test                                                                                                                                                                             
[ERROR] Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! - in junitplatform.FlakyFirstTimeTest
[ERROR] junitplatform.FlakyFirstTimeTest.testFailingTestOne  Time elapsed: 0.001 s  <<< FAILURE!                                        
org.opentest4j.AssertionFailedError: Failing test                                                                                                                                        
        at junitplatform.FlakyFirstTimeTest.testFailingTestOne(FlakyFirstTimeTest.java:42)                                                                                               
[ERROR] junitplatform.FlakyFirstTimeTest.testErrorTestOne  Time elapsed: 0.001 s  <<< ERROR!                                                                                             
java.lang.IllegalArgumentException: ...                                                                                                                                                  
        at junitplatform.FlakyFirstTimeTest.testErrorTestOne(FlakyFirstTimeTest.java:54)                                                                                                 
[INFO] Running junitplatform.PassingTest                                                                                                                                                 
Passing test one                                                                                                                                                                         
Passing test two                                                                                                                                                                         
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in junitplatform.PassingTest

Could you give me a hint why this happens / how to solely call my parameterized example tests?

[jira-importer]

Tibor Digana commented

DaGeRe1989
Excellent!
So you have coped with all these troubles and you wrote the test.
Now you can post the pullrequest on GH.
Acctually, I have expect a test error. This would drive us to:

• write the test
• fail the test as expected
• understand the problem
• and finally fix the problem

We are in 50% of the entire work, but that's pretty normal.

[jira-importer]

David Georg Reichelt commented

tibordigana 

Thanks for the quick reply.

I was a little surprised that I can post my pullrequest on GH now, since the test currently doesn't really work (it only calls test, without a selector, and it does not check whether certain tests are selected). Nevertheless, I created the pullrequest: #476

Hope this helps. If you need further input from me, tell me.