无论我做什么，都无法使用预签名 URL 将文件上传到 S3。 AWS 命令行有效。 CURL 和其他任何东西 = 403
Can't upload a file to S3 with pre-signed URL no matter what I do. AWS command line works. CURL and anything else = 403
我在 ~/.aws/credentials 中的 AWS 凭据正确且有效。证明？
$ aws s3api put-object --bucket <my bucket name> --key videos/uploads/yoda.jpeg --body /Users/r<my_name>/Desktop/Archive/yoda.jpeg
回来：
{
"ETag": "\"66bee0b7caf3d127900e0a70f2da4b5f\""
}
从命令行上传成功。当我在 AWS 的管理控制台中看到我的 S3 存储桶时，我可以看到我的文件。
现在-我从 S3 中删除了成功上传的文件，然后尝试再次上传，这次是通过预签名 URL
$ aws s3 presign s3://<my-bucket>/videos/uploads/yoda.jpeg
为此我得到：
https://<my-bucket>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T222859Z&X-Amz-Signature=3a3624b9e264c119ebdf93c989efb73337f7ab8793e89554c7b000e1fc93c85c
从这一刻起，任何 PUT 尝试使用 CURL、POSTMAN 或任何其他工具，使用此 URL 上传文件失败总是以403（是的，它没有过期，它立即失败）并且The request signature we calculated does not match the signature you provided是AWS提供的借口。
S3 存储桶有一个策略，允许凭据在该存储桶上 /.aws/credentials 到 Put* 的用户。
这是怎么回事？为什么预签名 URL 不起作用？
CURL 尝试
$ curl --location --request PUT 'https://<my-bucket-name>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T224403Z&X-Amz-Signature=8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670%0A' \
--header 'Content-Type: image/jpeg' \
--data-binary '@/Users/<MY-NAME>/Desktop/Archive/yoda.jpeg'
我得到：
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId><---MY--ACCESS--KEY--ID--->/AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20210207T224403Z
20210207/us-east-2/s3/aws4_request
da93cc1a0ec196fe0726ec6d5cace8c1b2b4865b20663bf0240454e276dbef6f</StringToSign>
<SignatureProvided>8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670
</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 0a 32 30 32 31 30 32 30 37 2f 75 73 2d 65 61 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 61 39 33 63 63 31 61 30 65 63 31 39 36 66 65 30 37 32 36 65 63 36 64 35 63 61 63 65 38 63 31 62 32 62 34 38 36 35 62 32 30 36 36 33 62 66 30 32 34 30 34 35 34 65 32 37 36 64 62 65 66 36 66</StringToSignBytes>
<CanonicalRequest>PUT
/videos/uploads/yoda.jpeg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<---MY--ACCESS--KEY--ID--->%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T224403Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host
host:<my-bucket-name>.s3.amazonaws.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>50 55 54 0a 2f 76 69 64 65 6f 73 2f 75 70 6c 6f 61 64 73 2f 79 6f 64 61 2e 6a 70 65 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 51 33 44 34 36 52 4e 50 48 51 4e 4b 47 42 46 4b 25 32 46 32 30 32 31 30 32 30 37 25 32 46 75 73 2d 65 61 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 6c 73 74 76 32 2d 70 75 62 6c 69 63 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>CBJT0Y4SX9A7RB26</RequestId>
<HostId>h+5b/u8cdi34yuSDBX0Z/mZGQMtRZIMS4rvIwiKzOZSOZhRoQfak8cOdVBq2BgtU1qbqlHrO2TY=</HostId>
</Error>
正在尝试从 PYTHON 生成预标记 URL。仍然不起作用。 URL 是错误的 - AWS 拒绝了相同的 403
def get_upload_pre_signed_url(bucket_name, object_name, expiration=3600):
s3_client = boto3.client('s3')
try:
response = s3_client.generate_presigned_url('put_object',
Params={'Bucket': bucket_name,
'Key': object_name},
ExpiresIn=expiration)
except ClientError as e:
return None
# The response contains the presigned URL
return response
URL由此生成：
https://<my-bucket>.s3.amazonaws.com//videos/uploads/yoda.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<my-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T231306Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=968a3e2cab9b7e907df69e24aae24d79ea40f52a52d407591d7cbd69c86fe67b
Curling 它得到相同的 403。没有改变。
aws s3 presign 命令创建可用于下载文件的 URL。它不会创建可用于上传的 URL。引用文档-
Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.
要创建上传 URL，您需要跳出命令行并进入您选择的语言并使用完整的 AWS SDK。
Python 示例：
s3.generate_presigned_post(
Bucket=BUCKET_NAME,
Key=FILE_KEY,
ExpiresIn=(5*60)
)
请注意，它使用 generate_presigned_post 函数来执行此操作。
s3 presign 仅用于生成 url 供下载
Generate a pre-signed URL for an Amazon S3 object. This allows anyone
who receives the pre-signed URL to retrieve the S3 object with an HTTP
GET request. All presigned URL’s now use sigv4 so the region needs to
be configured explicitly
我们仍然需要指定要上传到的 Bucket 和 Key。
节点：
const bucketParms = {
Bucket: "sample-temp-bucket",
Key: "HeadShot.jpg",
ContentType:'image/jpeg'
};
s3.getSignedUrl("putObject", bucketParms, (error, url) => {
if (error) console.log("error", error);
if (url) console.log("url", url);
});
Python:
response = s3_client.generate_presigned_url('put_object',
Params={'Bucket': bucket_name,
'Key': object_name,
'ContentType':'image/jpeg'},
ExpiresIn=expiration)
我们可以做一个 curl 或者 postman
curl --location --request PUT 'https://test-events.s3.amazonaws.com/98..?X....' \
--header 'Content-Type: image/jpeg' \
--data-binary '/Users/user/image/path'
presign cli 命令仅适用于 GET 请求。如果您需要其他任何东西，您必须直接使用 AWS API - 如下所示：您可以为此使用一个简短的 python 脚本。我们为我们的一个应用程序使用了一个 lambda，您可以调用它，您将获得正确的 url。此外，预签名 URL 使用调用 API 的角色，因此它具有相同的权限。包括以下事实：如果您正在使用 STS 担任角色并且授权在预签名 url 的到期时间之前到期，url 仍然会失败。但是如果你使用常规角色（比如你的 aws cli 配置文件），应该没问题。
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html
Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.
AWS CLI doesn't support presigned PUT URL yet. You can easily generate one using Python Boto3 though. The documentation is here. If you want a presigned PUT, you just need to let ClientMethod param be put_object.
您的编辑对于使用 SDK 生成 URL 是正确的。
也就是说，要使用 URL，HTTP headers curl 使用需要完全正确。值得注意的是，签名要求没有 Content-Type header 发送到服务器。由于 --data-binary 强制一个，我知道 get curl 做正确事情的最简单方法是使用 --upload-file 标志：
$ curl $URL --upload-file yoda.jpg
它对我也不起作用，它返回 403 forbidden。有帮助的事情是：
- 正在将签名设置为 v4
new AWS.S3({ signatureVersion: "v4" })
- 如果您使用元数据，请对内容进行编码
encodeURI("meta text with symbols#@");
这里更耦合ts/js代码：
const s3 = new AWS.S3({ signatureVersion: "v4" });
const s3Params = {
Bucket: <BUCKET_NAME>,
Key: <KEY>,
Expires: 300,
ContentType: <CONTENT_TYPE>,
Metadata: {
title: encodeURI('SomeTitle with utf-8 characters'),
},
};
const signedPutUrl = await s3.getSignedUrlPromise(
"putObject",
s3Params
);
console.log('the signed url',signedPutUrl);
return {'url':signedPutUrl};
解决方案：原来我使用的 Boto3 不是最新的，我用错了。修复这些问题后，对我有用的代码是：
# THE CREDENTIALS ARE PART OF MY TESTING CODE. NO WORRIES THEY'RE IN AN ENV VARIABLE NOW
def get_upload_pre_signed_url(bucket_name, key, expiration=3600):
s3 = boto3.client('s3',
aws_access_key_id="<my access_key_id",
aws_secret_access_key="<my_secreet_access_key>",
config=Config(region_name='us-east-2', s3.{"use_accelerate_endpoint": True}))
try:
url = s3.generate_presigned_url('put_object', Params={'Bucket': bucket_name, 'Key': key},
ExpiresIn=expiration,
HttpMethod='PUT')
except ClientError as e:
return None
return url
我在 ~/.aws/credentials 中的 AWS 凭据正确且有效。证明？
$ aws s3api put-object --bucket <my bucket name> --key videos/uploads/yoda.jpeg --body /Users/r<my_name>/Desktop/Archive/yoda.jpeg
回来：
{
"ETag": "\"66bee0b7caf3d127900e0a70f2da4b5f\""
}
从命令行上传成功。当我在 AWS 的管理控制台中看到我的 S3 存储桶时，我可以看到我的文件。
现在-我从 S3 中删除了成功上传的文件，然后尝试再次上传，这次是通过预签名 URL
$ aws s3 presign s3://<my-bucket>/videos/uploads/yoda.jpeg
为此我得到：
https://<my-bucket>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T222859Z&X-Amz-Signature=3a3624b9e264c119ebdf93c989efb73337f7ab8793e89554c7b000e1fc93c85c
从这一刻起，任何 PUT 尝试使用 CURL、POSTMAN 或任何其他工具，使用此 URL 上传文件失败总是以403（是的，它没有过期，它立即失败）并且The request signature we calculated does not match the signature you provided是AWS提供的借口。
S3 存储桶有一个策略，允许凭据在该存储桶上 /.aws/credentials 到 Put* 的用户。
这是怎么回事？为什么预签名 URL 不起作用？
CURL 尝试
$ curl --location --request PUT 'https://<my-bucket-name>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T224403Z&X-Amz-Signature=8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670%0A' \
--header 'Content-Type: image/jpeg' \
--data-binary '@/Users/<MY-NAME>/Desktop/Archive/yoda.jpeg'
我得到：
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId><---MY--ACCESS--KEY--ID--->/AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20210207T224403Z
20210207/us-east-2/s3/aws4_request
da93cc1a0ec196fe0726ec6d5cace8c1b2b4865b20663bf0240454e276dbef6f</StringToSign>
<SignatureProvided>8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670
</SignatureProvided>
<StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 0a 32 30 32 31 30 32 30 37 2f 75 73 2d 65 61 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 61 39 33 63 63 31 61 30 65 63 31 39 36 66 65 30 37 32 36 65 63 36 64 35 63 61 63 65 38 63 31 62 32 62 34 38 36 35 62 32 30 36 36 33 62 66 30 32 34 30 34 35 34 65 32 37 36 64 62 65 66 36 66</StringToSignBytes>
<CanonicalRequest>PUT
/videos/uploads/yoda.jpeg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<---MY--ACCESS--KEY--ID--->%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T224403Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host
host:<my-bucket-name>.s3.amazonaws.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>50 55 54 0a 2f 76 69 64 65 6f 73 2f 75 70 6c 6f 61 64 73 2f 79 6f 64 61 2e 6a 70 65 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 51 33 44 34 36 52 4e 50 48 51 4e 4b 47 42 46 4b 25 32 46 32 30 32 31 30 32 30 37 25 32 46 75 73 2d 65 61 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 6c 73 74 76 32 2d 70 75 62 6c 69 63 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
<RequestId>CBJT0Y4SX9A7RB26</RequestId>
<HostId>h+5b/u8cdi34yuSDBX0Z/mZGQMtRZIMS4rvIwiKzOZSOZhRoQfak8cOdVBq2BgtU1qbqlHrO2TY=</HostId>
</Error>
正在尝试从 PYTHON 生成预标记 URL。仍然不起作用。 URL 是错误的 - AWS 拒绝了相同的 403
def get_upload_pre_signed_url(bucket_name, object_name, expiration=3600):
s3_client = boto3.client('s3')
try:
response = s3_client.generate_presigned_url('put_object',
Params={'Bucket': bucket_name,
'Key': object_name},
ExpiresIn=expiration)
except ClientError as e:
return None
# The response contains the presigned URL
return response
URL由此生成：
https://<my-bucket>.s3.amazonaws.com//videos/uploads/yoda.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<my-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T231306Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=968a3e2cab9b7e907df69e24aae24d79ea40f52a52d407591d7cbd69c86fe67b
Curling 它得到相同的 403。没有改变。
aws s3 presign 命令创建可用于下载文件的 URL。它不会创建可用于上传的 URL。引用文档-
Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.
要创建上传 URL，您需要跳出命令行并进入您选择的语言并使用完整的 AWS SDK。
Python 示例：
s3.generate_presigned_post(
Bucket=BUCKET_NAME,
Key=FILE_KEY,
ExpiresIn=(5*60)
)
请注意，它使用 generate_presigned_post 函数来执行此操作。
s3 presign 仅用于生成 url 供下载
Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly
我们仍然需要指定要上传到的 Bucket 和 Key。
节点：
const bucketParms = {
Bucket: "sample-temp-bucket",
Key: "HeadShot.jpg",
ContentType:'image/jpeg'
};
s3.getSignedUrl("putObject", bucketParms, (error, url) => {
if (error) console.log("error", error);
if (url) console.log("url", url);
});
Python:
response = s3_client.generate_presigned_url('put_object',
Params={'Bucket': bucket_name,
'Key': object_name,
'ContentType':'image/jpeg'},
ExpiresIn=expiration)
我们可以做一个 curl 或者 postman
curl --location --request PUT 'https://test-events.s3.amazonaws.com/98..?X....' \
--header 'Content-Type: image/jpeg' \
--data-binary '/Users/user/image/path'
presign cli 命令仅适用于 GET 请求。如果您需要其他任何东西，您必须直接使用 AWS API - 如下所示：您可以为此使用一个简短的 python 脚本。我们为我们的一个应用程序使用了一个 lambda，您可以调用它，您将获得正确的 url。此外，预签名 URL 使用调用 API 的角色，因此它具有相同的权限。包括以下事实：如果您正在使用 STS 担任角色并且授权在预签名 url 的到期时间之前到期，url 仍然会失败。但是如果你使用常规角色（比如你的 aws cli 配置文件），应该没问题。
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html
Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.
AWS CLI doesn't support presigned PUT URL yet. You can easily generate one using Python Boto3 though. The documentation is here. If you want a presigned PUT, you just need to let ClientMethod param be put_object.
您的编辑对于使用 SDK 生成 URL 是正确的。
也就是说，要使用 URL，HTTP headers curl 使用需要完全正确。值得注意的是，签名要求没有 Content-Type header 发送到服务器。由于 --data-binary 强制一个，我知道 get curl 做正确事情的最简单方法是使用 --upload-file 标志：
$ curl $URL --upload-file yoda.jpg
它对我也不起作用，它返回 403 forbidden。有帮助的事情是：
- 正在将签名设置为 v4
new AWS.S3({ signatureVersion: "v4" }) - 如果您使用元数据，请对内容进行编码
encodeURI("meta text with symbols#@");
这里更耦合ts/js代码：
const s3 = new AWS.S3({ signatureVersion: "v4" });
const s3Params = {
Bucket: <BUCKET_NAME>,
Key: <KEY>,
Expires: 300,
ContentType: <CONTENT_TYPE>,
Metadata: {
title: encodeURI('SomeTitle with utf-8 characters'),
},
};
const signedPutUrl = await s3.getSignedUrlPromise(
"putObject",
s3Params
);
console.log('the signed url',signedPutUrl);
return {'url':signedPutUrl};
解决方案：原来我使用的 Boto3 不是最新的，我用错了。修复这些问题后，对我有用的代码是：
# THE CREDENTIALS ARE PART OF MY TESTING CODE. NO WORRIES THEY'RE IN AN ENV VARIABLE NOW
def get_upload_pre_signed_url(bucket_name, key, expiration=3600):
s3 = boto3.client('s3',
aws_access_key_id="<my access_key_id",
aws_secret_access_key="<my_secreet_access_key>",
config=Config(region_name='us-east-2', s3.{"use_accelerate_endpoint": True}))
try:
url = s3.generate_presigned_url('put_object', Params={'Bucket': bucket_name, 'Key': key},
ExpiresIn=expiration,
HttpMethod='PUT')
except ClientError as e:
return None
return url