By default Ice fails to use S3 on eu-central-1 #215
Description
I configured all the things in AWS as explained in the README of Ice and as we are mostly (99%) using AWS from eu-central-1, I setup the S3 bucket to hold the billing reports on this datacenter.
After starting up and download the instances prices and everything, it successfully manages to download some things from this bucket on eu-central-1 but eventually fails with the following error:
ice_1 | 2016-05-30 13:41:04,902 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download 123456789-aws-billing-detailed-line-items-2016-04.csv.zip...
ice_1 | | Error 2016-05-30 13:41:05,068 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error polling
ice_1 | Message: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: XXXXXXXXXXXXX)
ice_1 | Line | Method
ice_1 | ->> 1077 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient
ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ice_1 | | 725 | executeOneRequest in ''
ice_1 | | 460 | executeHelper . . . . . . in ''
ice_1 | | 295 | execute in ''
ice_1 | | 3710 | invoke . . . . . . . . . . in com.amazonaws.services.s3.AmazonS3Client
ice_1 | | 1010 | getObjectMetadata in ''
ice_1 | | 988 | getObjectMetadata . . . . in ''
ice_1 | | 275 | downloadFileIfChangedSince in com.netflix.ice.common.AwsUtils
ice_1 | | 175 | poll . . . . . . . . . . . in com.netflix.ice.processor.BillingFileProcessor
ice_1 | | 50 | doWork in com.netflix.ice.common.Poller
ice_1 | | 28 | access$000 . . . . . . . . in ''
ice_1 | | 88 | run in com.netflix.ice.common.Poller$1
ice_1 | ^ 745 | run . . . . . . . . . . . in java.lang.Thread
(running from Docker Compose)
It was hard to find more information, but I managed to upgrade the log levels, saw the actual HTTP HEAD query made and tried to do the same with a GET instead and I got the following error message instead:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidRequest</Code>
<Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message>
</Error>
For some reason, this mechanism is not used directly and I guess that's why it fails on eu-central-1 which doesn't support other mechanism (as far as I understand).
It looks like #193 is here to address this problem although both the explanation and the fix are confusing.
After looking more closely at the code, it appears that if I set the EC2_REGION system property to eu-central-1 at startup it solves the problem, but:
- This is not documented anywhere;
- I'm pretty sure there's a way to get the authentication mechanism right, even it the user doesn't manually specify the S3 region to use (or that's what the AWS SDK promise seems to be?)
It would be cool if at least the first point could be addressed so users using other regions than the US ones are also able to start Ice, although I'm not sure exactly why this setting was introduced in the first place.
Probably a better fix would be to use the GetLocation method on the bucket to get where the location of the bucket is and set the correct endpoint then.