HTTP Method overriden on Spring-boot 2.7.8 but the 3 first letter of the first cookie

[jfisbein]

I just upgraded one application from spring-boot 2.7.7 to 2.7.8 and then every HTTP request including a cookie started to fail.
After some investigation, we found that the HTTP method is being substituted by the 3 first letters of the first cookie name (I know it sounds crazy).

Here is my request:

curl -v 'http://localhost:8080' \
  -H 'Accept: text/html' \
  -H 'Accept-Language: en-US,en;q=0.9,es;q=0.8' \
  -H 'Cookie: hDz_user_id=asdfa'

And here is the stack trace of the exception:

Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the HTTP method "hDz" was not included within the list of allowed HTTP methods [HEAD, DELETE, POST, GET, OPTIONS, PATCH, PUT]
	at org.springframework.security.web.firewall.StrictHttpFirewall.rejectForbiddenHttpMethod(StrictHttpFirewall.java:527)
	at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:504)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:206)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at datadog.trace.instrumentation.springweb.HandlerMappingResourceNameFilter.doFilterInternal(HandlerMappingResourceNameFilter.java:50)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)

Has you can see the exception is complaining about method "hDz", which is the first 3 letters of the cookie, I made some tests with different cookie names and I always got an error about the method being the first 3 letters of the cookie name.

If I downgrade to spring-boot 2.7.7 the error disappears and everything starts working again.

[bclozel]

As discussed in the blog post comments, this looks similar to #33735 - is there a Java agent involved?

[jfisbein]

Yes, Datadog agent 0.115.0

[bclozel]

Can you reproduce the problem without the datadog agent?

[jfisbein]

I just updated the datadog agent to the latest version (1.5.0) and everything works.
Thank you for the support.

[ljrmorgan]

I had a semi-related sounding issue when updating to from 2.7.7 to 2.7.8 that was also caused by a Java agent. I thought I would mention it here in case anyone else hits the same problem.

In our case the DispatcherServlet didn't seem to start up at all (no Initializing Servlet 'dispatcherServlet' log line in our logs) and all requests to my app (including to actuator endpoints) would return 404s with an HTML response body from Tomcat. Updating from 1.18.0 to 1.22.1 of the OpenTelemetry Java agent resolved it for us.

[wllianwd]

But whats is the reason that this issue just happen when Spring Boot is upgraded from 2.7.7 to 2.7.8?

[bclozel]

@wllianwd as seen in this comment, this is probably not linked to Spring Boot itself, but rather the Tomcat version upgrade. It could be that the Java agent relies on Tomcat internals (like the ordering of parameters for an internal method) and is broken by an internal change in Tomcat.

[trask]

It could be that the Java agent relies on Tomcat internals (like the ordering of parameters for an internal method) and is broken by an internal change in Tomcat.

sorry for the late comment, just wanted to confirm that this is correct

the OpenTelemetry Java agent has a safety feature that prevents applying instrumentation when a mismatch between the instrumentation code and the instrumented application code is detected

unfortunately, in this case there was a behavioral change in one of the internal Tomcat APIs that the Java agent relies on, but no signature change, and so this mechanism wasn't triggered

the OTel Java agent's automated "latest dependency" tests did pick this issue up right away the day the new Tomcat / Spring Boot versions were released, and the issue was immediately fixed in the OpenTelemetry Java agent

unfortunately though, many users still ran into this issue when they deployed a newer version of Spring Boot / Tomcat along with an older version of the OpenTelemetry Java agent