400 Bad Request when using SSL and SAN certificates

[kwisatz]

We have our graylog instance sitting on a server that is not directly reachable from the Internet. In order to allow access to the HTTP GELF input, we defined a ProxyPass on a publicly visible server like so:

    <Location /gelf/>
        ProxyPass http://10.24.0.42:12201/gelf
        ProxyPassReverse http://10.24.0.42:12201/gelf
    </Location>

However, trying to establish a connection to graylog through this using the stream_socket_client as the StreamSocketClient does, always results in a 400 - Bad Request response by the Apache server:

RuntimeException: Graylog-Server didn't answer properly, expected 'HTTP/1.x 202 Accepted', response is 'HTTP/1.1 400 Bad Request Date: Thu, 22 Sep 2016 17:44:48 GMT Server: Apache/2.4.10 (Debian) Content-Length: 445 Connection: close Content-Type: text/html; charset=iso-8859-1' in /var/www/vendor/graylog2/gelf-php/src/Gelf/Transport/HttpTransport.php on line 189

Using curl, there's no issue at all:

kwisatz@mazer:~$ curl -i -XPOST https://graylog.domain.tld/gelf/ -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'
HTTP/1.1 202 Accepted
Date: Thu, 22 Sep 2016 17:15:07 GMT
Server: Apache/2.4.10 (Debian)
Content-Length: 0
Strict-Transport-Security: max-age=15768000
Connection: close

I've been trying to figure out if I had to configure mod_proxy differently for this to work but haven't really found any useful clues. I'm also not familiar with PHP's stream_socket_client and how this works on ssl:// socket connections.

It works just fine when not using SSL, but I'd really prefer not to send this data over the wire in plain-text.

I was hoping someone might have had the same issue already and solved it or someone might know how to configure mod_proxy with Apache?

Or, failure to figure this out, would it be possible to switch out the socketClient for a simple php-curl client?

[kwisatz]

To briefly follow up on this, we're seeing the exact same behavior with nginx:

RuntimeException: Graylog-Server didn't answer properly, expected 'HTTP/1.x 202 Accepted', response is 'HTTP/1.1 400 Bad Request Server: nginx Date: Sat, 24 Sep 2016 14:15:13 GMT Content-Type: text/html Content-Length: 264 Connection: close' in /var/www/vendor/graylog2/gelf-php/src/Gelf/Transport/HttpTransport.php on line 189

Here's the config for that reverse proxy instance:

server {
    listen 443;
    server_name graylog2.domain.tld;

        include standard-ssl;

    location /gelf {
        proxy_pass http://10.24.0.42:12201/gelf;
        proxy_read_timeout 120;
            proxy_connect_timeout 120;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }   
}

Here too it works when running it on port 80 using the tcp:// protocol for the socket client.

[kwisatz]

OK, so there seem to be a couple of issues at play here:

1. When creating a HttpTransport using the constructor and not passing $sslOptions, those are not initialized. So using port 443 is not enough to have the transport initialize $sslOptions. One has to either create and pass $sslOptions one-self or use HttpTransport::fromUrl() to have the transport use an SSL context.
2. There's a bug in PHP that prevents the stream_socket_client from looking at an SSL certificate's Subject Alternative Names (SANs):
   The result of that is a warning message like the following if the domain of your graylog server is not the first domain mentioned in your certificate. See http://blog.debug.cz/2012/11/https-connections-with-zend-framework-2.html. Interestingly, this was apparently fixed in 2013, but I still see the problem? (Utilize SAN extension matching during SSL/TLS peer verification php/php-src#482)

Warning: stream_socket_client(): Peer certificate CN=`first.tld' did not match expected CN=`graylog2.domain.tld' in \
 /var/www/vendor/graylog2/gelf-php/src/Gelf/Transport/StreamSocketClient.php on line 153

To "fix" this, one can use $sslOptions->setVerifyPeer(false); but that really is not what we should do.

An alternative fix could be to modify how the $sslOptions work. Instead of having setVerifyPeer expect a boolean, one could allow it to be a string, thus allowing us to set the CN_match value through it by further modifying getContext() in HttpTransport as follows:

// If verify_peer is active we also check for a valid CN
if ($this->sslOptions->getVerifyPeer()) {
     $context['ssl']['CN_match'] = $this->sslOptions->getVerifyPeer();
}

For that to work, one needs to build the $sslOptions and HttpTransport object like this:

$sslOptions = new Gelf\Transport\SslOptions();
$sslOptions->setCaFile('/etc/ssl/certs/ca-certificates.crt');
$sslOptions->setVerifyPeer('first.tld');
$transport = new Gelf\Transport\HttpTransport(
                'graylog2.domain.tld',
                443,
                '/gelf',
                $sslOptions
            );

[kwisatz]

All right, so I figured I had an outdated version of PHP, which would be why the patch mentioned in php/php-src#482 wouldn't work for me… sigh

So thanks for bearing with me while I figured this out and filled your notification box.

The only thing that remains in this case are these deprecation warnings:

Deprecated: SNI_server_name is deprecated in favor of peer_name in /var/www/vendor/graylog2/gelf-php/src/Gelf/Transport/StreamSocketClient.php on line 153

Deprecated: the 'CN_match' SSL context option is deprecated in favor of 'peer_name' in /var/www/vendor/graylog2/gelf-php/src/Gelf/Transport/StreamSocketClient.php on line 153

[bzikarsky]

Sorry for the late response. I was on vacation during your analysis. 😄

It's very nice that you were able to identify and solve the problem by yourself. Though I'd like to fix the SSL deprecation warnings. I will create a separate issue for this.

Thanks for your time!

[kwisatz]

Thanks and you're welcome.
I was actually planning on adding what I learned to the README, but haven't gotten around to it yet.