Add ability to execute a subset of rules from the via system properties

[bobtiernay-okta]

When developing large suite of ArchUnit rules, one common pattern is using multiple ArchRules / ArchTests in a single test class file:

 @AnalyzeClasses(..)
   class MyArchitectureTest {
       // includes all @ArchTest members from MyArchRuleSuite1
       @ArchTest
       static final ArchTests includedRules1 = ArchTests.in(MyArchRuleSuite1.class);
  
       // includes all @ArchTest members from MyArchRuleSuite2
       @ArchTest
       static final ArchTests includedRules2 = ArchTests.in(MyArchRuleSuite2.class);
}

However, each included ArchTests may run very slowly and it appears as though it is not possible to only run a subset of these tests from the command line. It would be great if new system properties were exposed that allowed JUnit 4 / 5 runners to only execute rules by pattern, in similar fashion to how JUnit does for test classes. Ideally this would work well with existing tooling for Maven (e.g. Surefire) and Gradle (e.g. test filtering).

One idea is exposing 2 new system properties:

• archunit.tests.includePattern
• archunit.tests.excludePattern

Which support wildcards. e.g.

• my.org.MyArchitectureTest.includedRules1.rule2
• *MyArchitectureTest.includedRules*.rule*

[codecholeric]

Thanks for bringing this idea! 😃 I agree that this is one annoying part at the moment, personally I also comment the @ArchTest I don't want to execute 🤪 In theory there would be an API of archunit-junit5-engine that would allow to run single fields just like single methods. But it would likely need a custom IDE / build tool plugin. I would guess from Gradle it would even be possible to patch this in somehow, but it's certainly not easy to use at the moment 😞

[crizzis]

I'm not sure I follow. What is wrong with the existing tooling support? Is it not possible to exclude test classes by category / file name?

Also, what's the point of grouping ArchUnit suites under one uber-suite if you're going to then exclude some of them anyway? How was the grouping helpful in the first place, then?

[codecholeric]

One common way I have used several times is to group rules together and include them hierarchically, like

@AnalyzeClasses(..)
class ProjectOneArchRules {
    @ArchTest
    static final ArchTests serviceRules = ArchTests.in(ServiceRules.class);

    @ArchTest
    static final ArchTests controllerRules = ArchTests.in(ControllerRules.class);
}

This works completely fine in CI where I want to continuously execute that whole suite. But now let's say I'm confused why controller rule 5 is failing suddenly (let's assume the message is not self-explanatory this time 😉). So now I want to execute only controller rule 5 locally to test it real quick. At the moment this is really tedious to achieve. That's what this issue is about.

[crizzis]

@codecholeric Please have a look at my draft PR when you have a moment.

The PR:

• introduces support for JUnit 4's Assume API
• introduces support for Jupiter's @EnabledIfXxx annotations (for method-flavored tests) and a set of bridging @Conditions.EnabledIfXxx annotations for field-based tests
• introduces support for Jupiter's @Tag annotation for method-flavored tests
• introduces support for Jupiter's @Disabled for method-flavored tests
• enables Surefire test filtering for field-flavored tests (for JUnit Jupiter and JUnit Vintage, but not vanilla JUnit 4, which I believe is a fair tradeoff); note that modifications to Surefire are also needed - see related Jira issue
• enables Gradle test filtering for field-flavored tests, which now seems to work for both JUnit 4 and JUnit 5; note that modifications to Gradle are also needed - see related GitHub issue, which I doubt is the final version of the solution

The reception of the related PR at Surefire was rather frosty, and someone from Gradle commented that they have other priorities at the moment. After closer inspection of the code, though, it seems it would be enough to make FieldSource extend MethodSource, returning the field name from getMethodName, so if things go south with either Surefire or Gradle, this seems like a viable hac*... ekhrm workaround.

One last thing, the PR is a draft mostly because it relies on custom versions of Surefire and Gradle. Once both PRs are merged, the dependency on the custom Surefire version can be lifted. Not entirely sure what to do about the dependency on Gradle, because archunit-junit5-engine now depends on gradle-testing-junit-platform, and Gradle seems to have stopped publishing individual Jars to public repos recently.

[hjohn]

This seems to support JUnit 5 much better than the current version.

I'm having major trouble providing a set of common rules for our projects, but excluding some if a certain project doesn't like one of the standard rules. ArchTests.in doesn't cut it because it does not allow excluding rules, nor does it facilitate having a set of common rules that can be extended without explicitly having to include them in a project.

At a minimum it would be good if ArchUnit would work properly with non-static methods, because then the (abstract) base class could do:

boolean packagesShouldBeFreeOfCyclesRuleActivated = true;

@ArchTest
public void packagesShouldBeFreeOfCycles(JavaClasses classes) {
    assume(packagesShouldBeFreeOfCyclesRuleActivated);

    slices().matching(BASE_PACKAGE_NAME + ".(**)").should().beFreeOfCycles().check(classes);
}

And subclasses can then turn this off if they don't like this standard rule.

Unfortunately, instantiation fails.

JUnit 5 offers tons of options to control which tests to run (Tags, EnabledIf/DisabledIf annotations, Assumptions, TestFactory) but basically in ArchUnit's current form none of them can be used for this use case.

[codecholeric]

The problem is that ArchUnit's TestEngine should not have any dependency on the Jupiter API (which is actually a big problem with #834). The Jupiter API doesn't fully match the model (e.g. fields can't be tests there, so some annotations can't be assigned to fields).

But we can of course implement a reasonable subset in a compatible way. One thing that already is possible though is tagging, which you can do via @ArchTag.

This issue originally was just about a simple test filter, so I feel it has been hijacked a little 🤔 I'm happy to collect possible features with some reasoning for which case they are needed, but we should discuss them first. In particular we should not reuse Jupiter APIs or add a dependency to it. And I would likely create separate issues then with concrete action items.

Maybe you could start by describing how exactly you would want to run your tests and where the current limitations prevent you from doing it in a good way (so we can ponder about the cleanest solution to solve it, I would e.g. avoid inheritance if possible).

[crizzis]

If I may drop in my two cents, the PR can be conceptually divided into two parts: (1) making sure test filtering works with popular test tools, and (2) better support for JUnit 5 and JUnit 4 test exclusion-related API.

TBH even while implementing both parts, I felt that their relative usefulness (given what facilities are already provided by ArchUnit API itself) is strongly in favour of (1).

@codecholeric Since you've expressed concerns over adding a dependency on JUnit 5 API, then perhaps we could cherry-pick the part dealing with (1), and move the discussion about the extent to which we want/need (2) to another issue?

I believe that would provide a way for users to solve 90% of possible use cases, and (2) could be moved to an optional pluggable module, if need be.

WDYT?

(Sorry for making a tangled mess of a PR which I now very clearly see should have been divided into at least two distinct parts, BTW)

[hjohn]

I can understand you don't want JUnit 5 specific stuff in ArchUnit itself, however, if we can just call into ArchUnit from a normal JUnit test (I've looked, but couldn't find a way) then you can use what JUnit already offers to control what tests are run. I feel that annotations like ArchTest and ArchTag are just reinventing what is already available in JUnit.

As for a description of what I think would good to have: I would like to offer a full set of ArchUnit rules in a common project. Projects interested in using those rules would simply create one JUnit test case that uses these common rules (either by subclassing a base class, or some other mechanism like ArchTests.in). They may need to provide some parameters (like base package name).

The problem however starts when a certain project isn't quite ready for the full set of rules, or when a certain rule is controversial. Such rules should be easily disabled. ArchUnit offers inclusion mechanisms only (as far as I could find), but no exclusion mechanism. This is problematic because if rules are added to the commons project they will not be automatically applied to projects importing the common rules (a build may fail, but then those rules can consciously be turned off or fixed). So I think what's needed is:

• A way to offer a set of common rules (imported from another artifact or project)
• A way to run all those rules without having to specify all of them so an upgrade of the common rules would run newly added rules
• A way to exclude some of those rules that projects are not ready for, are controversial or are too hard to fix

This could be solved in several ways, several which I've attempted but failed to get to work:

• Make non-static @ArchTest annotations work in a super class; when applied to a method it would allow to stop the test from running using assume on some condition dictated by a sub class
• ArchTag works similar to JUnit Tag which does not seem to offer a way to include or exclude tests programmatically
• Try calling ArchUnit from a regular @Test; I couldn't find a way to get the required JavaClasses object

Tags could definitely work, but what I'm missing is a way to do something like:

@AnalyzeClasses(packages = ArchitectureTest.BASE_PACKAGE_NAME)
@ArchExcludeTag({"namingConventions", "controversialPackageRule"})  // <--
public class ArchitectureTest {
    static final String BASE_PACKAGE_NAME = "com.acme.commons.core";

    @ArchTest
    private final ArchTests standardRules = ArchTests.in(StandardRules.class);
}