Problems basing documentation on MockMVC.

[davidgoate]

First of all, thanks for the work so far on this project. I heard about this on the recent Spring webinar and I was excited to give it a try. I'd like to suggest a possible enhancement, but, I'm not really sure whether this goes against the basic principle of this project.

When using Spring Boot (or for any situations where it's just as easy to get a full embedded container up and running) there is arguably no need for MockMVC, and I believe it actually can do harm.

I followed one of the sample projects using Spring Boot and initialise MockMVC like this:

    @Autowired
    private WebApplicationContext context;
    private MockMvc mockMvc;

    @Before
    public void setUp() {
        this.mockMvc = MockMvcBuilders.webAppContextSetup(this.context)
                .apply(new RestDocumentationConfigurer())
                .alwaysDo(document(getDocumentationName()))
                .build();
    }

However - and it could very easily be a lack of my understanding of MockMVC - I feel that there are issues with this:

• When creating a MockMVC instance from a fully built WebApplicationContext it appears that not all filters are copied (for example Spring Security filters).
• Other important configuration (such as the context path or some error handling logic) isn't used.

This means that if an app is always run on a path such as "/api/" e.g. "https://myhost/api/" there doesn't seem to be a way for the documentation to be aware of the "/api" path meaning all sample CURL requests won't work.

Furthermore, due to the lack of filters I have observed differences in actual runtime behaviour when compared with test behaviour in terms of response headers and error handling. E.g.

In tests if I do something like:

getMockMvc()
                .perform(get("/no-such-endpoint").accept(MediaType.APPLICATION_JSON))
                .andExpect(status().isNotFound());

I get simply

HTTP/1.1 404 Not Found

However, against the real app I get:

curl http://localhost:8080/api/no-such-endpoint -i -H "Accept: application/json"
HTTP/1.1 404 Not Found
Server: Apache-Coyote/1.1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
X-Application-Context: application:development
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Tue, 24 Mar 2015 15:11:05 GMT

{
  "timestamp" : 1427209865283,
  "status" : 404,
  "error" : "Not Found",
  "message" : "No message available",
  "path" : "/api/no-such-endpoint"
}

I appreciate that the reason that none of the Spring Security headers are there is because of the filters not being present. But, there are also other potentially important differences such as the lack of response body.

I feel that an important attribute of documenting the API is that the documentation is as accurate as possible. This is why I like the idea of documentation being produced by tests.

I don't fully understand why these differences happen, but my suggestion would be that if at all possible this project could provide a special RestTemplate, e.g. DocumentingRestTemplate that could be used to make requests (instead of MockMvc) against a real embedded container and this way the results should be more accurate since it is a real version of the app.

Any thoughts or comments welcome and thanks so much for the work so far!

[firefoxNX]

Can you take a look at http://stackoverflow.com/questions/14561235/spring-mvc-integration-tests-with-spring-security Rob Winch's answer?

[davidgoate]

@firefoxNX Hi, thanks for the reply. Yes this would fix that problem, however, this was just one aspect of my issue with this.

I don't think the context path or difference in error handling is down to the missing security filters?

It begins to feel like one has to understand all of the differences between MockMVC and a real WebAppContext so that you can begin to copy or inject the config into MockMVC yourself when you have a perfectly good full blown context sitting there doing nothing.

Don't get me wrong, I like the concept of where this is going and it is a great start given this is so young and not even at its first release yet. I just feel like in scenarios where you do have easy access to a container, isn't it easier to use this rather than mock something that is a bit like it - but not quite.

[rstoyanchev]

Spring Security has a more recent integration with Spring MVC Test. Take a look here https://spring.io/blog/2014/05/23/preview-spring-security-test-web-security.

This means that if an app is always run on a path such as "/api/" e.g. "https://myhost/api/" there doesn't seem to be a way for the documentation to be aware of the "/api" path meaning all sample CURL requests won't work.

There is a defaultRequest method on the MockMvcBuilder that you can use to set up any host and/or context path you'd like to appear. Yes it's not the same as running against an actual container but then again you'd have to generate your documentation against the actual host of your REST API, which isn't ideal either.

When using Spring Boot (or for any situations where it's just as easy to get a full embedded container up and running) there is arguably no need for MockMVC, and I believe it actually can do harm.

You have a point here although I believe this has more to do with the overall strategy for testing. Yes integration tests are the ultimate test of everything but there is a reason why unit tests have a place in an overall testing strategy.

Spring MVC Test falls somewhere in between integration and unit testing. It's another option in the tool chest. In fact the different builders (standalone vs webAppContext) provide multiple options on the scale between integration and unit testing -- loading the actual Spring configuration vs testing one controller at a time.

Full integration tests can be harder to reason about and debug as there are more moving pieces and it takes longer to execute which can really add up if you web layer does things that call to other services for example. Spring MVC Test can test the controller web layer in isolation. You can write more nuanced tests that verify more things on a more fine-grained level. It doesn't replace the need for integration tests but it has a role as part of an overall strategy.

I do think you have a very valuable point in that Spring Boot does quite a few things for web applications out of the box including the registration of Servlet filters, error handling, and many others. There is a much larger gap to fill when using Spring MVC Test in a Spring Boot application to understand what you should add manually. I think what we need is a Spring Boot specific MockMvcBuilder.

[davidgoate]

@rstoyanchev Thanks for the detailed response and all of the points you have made.

I think the defaultRequest method should do the trick so thanks for that. The only slight issue is, if I understand correctly, I have to do MockMvcRequestBuilders.request(method, uri) which means I need to add each endpoint I might test manually. There doesn't seem to be an obvious way for me to just say, "for any request issued by this MockMvc instance use this context path".

Just to address your point

but then again you'd have to generate your documentation against the actual host of your REST API, which isn't ideal either.

The idea here is that I would still use something like localhost:8080 but I would set a context path. With Spring Boot I use server.context-path=/api. There wouldn't be a need to necessarily produce CURL commands that point at the live environment on https://myhost/api. The important thing is that since this tool makes it so easy to produce sample CURL commands for the documentation it would be nice if I can ensure that the context path is set so that they can just be copy/pasted and run against a dev instance or even local machine. Perhaps this could even be added as an option to the org.springframework.restdocs.config.RestDocumentationConfigurer with a withPathPrefix method or similar so that and CURL commands include the path prefix.

👍 for the Spring Boot specific MockMvcBuilder I think that this would take away some of this pain.

[wilkinsona]

Thanks for the thoughtful feedback, @davidgoate. It looks like @rstoyanchev has answered much of it already. Thanks, Rossen.

There's another issue open where someone is looking for the curl snippets to work for an app that's behind a load balancer/proxy, i.e. they want to generated URLs to point to the load balancer and not directly to the app. That, too, requires the configuration of a prefix but one that's used in a very different way to your context path requirement. I'm sure that both can be satisfied but it'll take some thought to get the naming/structure right so that the two can co-exist.

Spring Boot's error handling is a little tricky when used with MVC Test as it relies on request forwarding which MVC Test doesn't fully support. However, I'm not sure that you actually want to document error responses that frequently and, when you do, you can make a request directly to /error to get the expected response. Take a look at the errorExample test in the samples to see how I've handled this.