Context not being reused in tests when MockBeans are used

[alebar]

Hi!

I would like to start a discussion about the mechanism of caching application contexts that are using @MockBeans across spring boot tests.

I was following this guide about testing the web layer: https://spring.io/guides/gs/testing-web/ and I was surprised that application context was build independently for every test class. I was hoping it will be cached to decrease test time, but I was wrong.

I've created small sample app to show this. Here is pull request: spring-attic/spring-boot-issues#56

There's one @Service (FooBarService) used by controller (FooBarApplication) in two methods. And two tests that have some expectations about the web layer (as indicated by the @WebMvcTest annotation). Those two tests work on the same context - the only difference is distinct mocks. But if you do mvn clean test you'll see the spring banner twice, which means that context for the web layer is created two times.

I've done some research and debugging and found that this is caused by including the @MockBean and @SpyBean definitions in the key for context cache (through the MockitoContextCustomizer that is a part of MergedContextConfiguration).

Well... After rethinking all of this I can understand that from technical point of view those contexts are not the same, because mock@1 and mock@2 are not equal. But still I feel that my assumption that context should be reused is what users could expect. Especially if we realize that @WebMvcTest is used to reduce the test time by not starting tomcat. Having this goal in mind it's hard to accept the extra time of repeated context building.

I'm not sure if such a change is a bug fixing or enhancement. I feel that this behaviour is bug but you can discuss it.

For now my workaround (or maybe it's official way to do this?) is to create an abstract test class with all @MockBeans definitions, something like:

@RunWith(SpringRunner.class)
@WebMvcTest
public abstract class AbstractTest {
    protected @MockBean FooBarService service;
}

public class FooTest extends AbstractTest {...}

But maybe it's better to consider an approach in which context is build once and mocks are replaced in beforeTestClass and afterTestClass methods of TestExecutionListener? And to avoid refreshing dependencies in every class referencing those mocked beans maybe it's ok to generate some beanToMockProxy object, that will be autowired in controllers and will hold a reference to mockito mock (created per test class), proxing all the methods to that mock? I don't know... I'm just thinking loud.

Cheers,
Alek

[philwebb]

Thanks for the detailed analysis. This is indeed a bug and it was introduced when commit a985a5c changed MockDefinition to use a ResolvableType.

The hashCode and equals method of MockDefinition should match if two different definitions are defined, but result in the same mock. The problem is that ResolvableType is actually using the source field for its hashCode method.

[philwebb]

@jhoeller do you think this is a Spring Framework bug? Looking at ResolvableType, I wonder if hashCode() and equals() should use this.typeProvider.getType() rather than getSource()? It feels odd that two ResolvableTypes that ultimately resolve to the same thing should not be considered equal.

[philwebb]

I've raised https://jira.spring.io/browse/SPR-14826

[wilkinsona]

This is fixed in the latest Spring Framework 4.3.4 snapshots. Overriding spring.version in the project that reproduced the problem results in this output:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running example.BarTest

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.1.RELEASE)

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.47 sec - in example.BarTest
Running example.FooTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in example.FooTest

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Note that the banner only appears once, i.e. the context has been reused.

[alebar]

Thank you all very much for taking care of that :)

I know that the issue is closed but I'd like to follow up on the topic of context caching. You've fixed the mechanism of calculating cache's keys based on mock definitions. But I'm not convinced for the general idea of including mocks in that key :(

Please consider one more example that I've provided in issues repo. The PR is here: spring-attic/spring-boot-issues#59

In that example I have two tests, that are testing separate features and are independent. And they are using the same WebMvc context - the only difference is in mocks used in both. Despite that, spring is creating two contexts (which may take long time) instead of one (run mvn test to see banner twice).

My feeling is that developer's decision of what to mock and when shouldn't have impact on context creation. I'd expect in that case one application context, shared between tests and with @MockBeans created and injected into context only for one test class.

What do you think?

[wilkinsona]

At the time when we're getting the context cache key to decide whether a context can be reused or we need to create a new one, we have no knowledge at all of how the context's beans will be wired together. We also have no knowledge of the functionality that your test is going to exercise. This means that it's impossible for us to know that your FooTest can happily run with a mocked BarService and that your BarTest can happily run with a mocked FooService.

[ninj]

Might be not quite be what @alebar intended, but would using a @ContextHierarchy help any?

http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/context/ContextHierarchy.html