Watch, Follow, &
Connect with Us

Please visit our new home
community.embarcadero.com.


Welcome, Guest
Guest Settings
Help

Thread: Impossible to multipart upload to Amazon S3 on anything but USClassic



Permlink Replies: 5 - Last Post: Aug 23, 2017 6:01 AM Last Post By: Walter Frederic...
Rodrigo Gomez C...

Posts: 32
Registered: 3/16/00
Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 4, 2016 2:45 PM
Hello,

I moved an app from XE5 to 10.1 Berlin and now I find that my S3 uploads aren't working.

If I try to upload a file to a bucket outside the US Classic / Us East 1 regions then I get this error:

StatusCode: 400 - Message: Bad Request - The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-central-1' (AuthorizationHeaderMalformed)


This message is generated when calling
TAmazonStorageService.InitiateMultipartUpload


If I specify a bucket which is in fact on us-east-1 then there are no problems.

I don't know if there is anything I am missing to configure or setup, to be able to use another region? With the program compiled on XE5 and using those same buckets there is no problem at all to use any bucket. The code is the same, the only change is the compiler version.

Looking at the source code for the AmazonAPI, it indeed appears that the request does not take into account, at all, the region: the host headers (later used to get what is the region) are always:
bucket_name.s3.amazonaws.com
. This from following the code from InitiateMultipartUploadXML, which in turns calls InitHeaders, and this calls GetConnectionInfo.VirtualHost (Data.Cloud.AmazonAPI.pas line 2404), which never includes the region.

I have tried changing the StorageEndpoint (which is used on VirtualHost) to include the region but then I get another error:
The request signature we calculated does not match the signature you provided. Check your key and signing method.


Right now I am stuck with this, but this is something so obvious as to be a bug that I think I am doing something wrong, but can't find what. I can't also find anything on the documentation that specifies what to do in case the region is not the default one.

Thanks in advance
Jose Luis Rocha

Posts: 77
Registered: 1/11/02
Re: Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 5, 2016 1:17 PM   in response to: Rodrigo Gomez C... in response to: Rodrigo Gomez C...
El 04/06/2016 a las 23:45, Rodrigo Gomez Cordova escribió:
I moved an app from XE5 to 10.1 Berlin and now I find that my S3 uploads aren't working.

Maybe related with RSP-12649

https://quality.embarcadero.com/browse/RSP-12649

--
J.L.Rocha [OfiPro]
Rodrigo Gomez C...

Posts: 32
Registered: 3/16/00
Re: Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 6, 2016 4:37 PM   in response to: Jose Luis Rocha in response to: Jose Luis Rocha
Jose Luis Rocha wrote:
El 04/06/2016 a las 23:45, Rodrigo Gomez Cordova escribió:
I moved an app from XE5 to 10.1 Berlin and now I find that my S3 uploads aren't working.

Maybe related with RSP-12649

https://quality.embarcadero.com/browse/RSP-12649

I'm not sure. I can make multipart uploads, using UploadPart, as long as I upload to a bucket in the US Classic region. The problem is that the new V4 signature includes the region, and the code that defines the region depends on the Host header, and that host header is never generated correctly (i.e., it doesn't include the correct host) for other regions.

I have tried, as I said, specifying the host directly, or modifying the EndPoint to include the correct subdomain, but then I get an incorrect signature. Somewhere it must not be taking into account those changes.

This seems so basic to me that I really believe I'm doing something wrong on my end, but I can't find anything. And the same code on XE5 works without problems.

I could try to create a test case and upload it to Embarcadero's Jira, but this would need specific amazon credentials/buckets so I'm not sure how could I proceed with this. In any case, it should be really easy to reproduce: just try uploading to a new bucket defined on some other region. If I remember correctly, buckets created before certain date can still use the previous signature version (which doesn't include the region) so it needs to be a recent bucket.
Jose Luis Rocha

Posts: 77
Registered: 1/11/02
Re: Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 7, 2016 5:47 AM   in response to: Rodrigo Gomez C... in response to: Rodrigo Gomez C...
El 07/06/2016 a las 1:37, Rodrigo Gomez Cordova escribió:
I can make multipart uploads, using UploadPart, as long as I upload to a bucket in the US Classic region.

I think Delphi is making multipart uploads with the previous signature
version. You can make multipart uploads in US Classic region because old
regions still accepts the previous protocol.

Acording to Amazon:

"Amazon S3 supports Signature Version 4, a protocol for authenticating
inbound API requests to AWS services, in all AWS regions. At this time,
AWS regions created before January 30, 2014 will continue to support the
previous protocol, Signature Version 2. Any new regions after January
30, 2014 will support only Signature Version 4 and therefore all
requests to those regions must be made with Signature Version 4."

Can you try a multipart upload in Ireland region, to confirm this ?
Ireland region still accepts the old protocol. You should be able to do
it without errors.

--
J.L.Rocha [OfiPro]

Rodrigo Gomez C...

Posts: 32
Registered: 3/16/00
Re: Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 10, 2016 10:17 AM   in response to: Jose Luis Rocha in response to: Jose Luis Rocha
Hello José Luis,

The problem is not the signature version. They are using the correct one (v4) but with incorrect data: They are always passing the US Classic region. I am not 100% sure that I already tried with the Ireland region, but I'm think I did, and with the same result.

In any case, I have reported the bug to Embarcadero's Quality portal, and the status is already opened. Hopefully they can fix this and produce a patch soon.

https://quality.embarcadero.com/browse/RSP-15127

Regards
Walter Frederic...

Posts: 1
Registered: 11/27/06
Re: Impossible to multipart upload to Amazon S3 on anything but USClassic
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 23, 2017 6:01 AM   in response to: Rodrigo Gomez C... in response to: Rodrigo Gomez C...
You only need to specify a StorageEndpoint on the Connection used, with the Endpoint correspondent to the used region.

Rodrigo Gomez Cordova wrote:
Hello José Luis,

The problem is not the signature version. They are using the correct one (v4) but with incorrect data: They are always passing the US Classic region. I am not 100% sure that I already tried with the Ireland region, but I'm think I did, and with the same result.

In any case, I have reported the bug to Embarcadero's Quality portal, and the status is already opened. Hopefully they can fix this and produce a patch soon.

https://quality.embarcadero.com/browse/RSP-15127

Regards
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02