Glue Boto 客户端——NoCredentialsError

Question

几个月来，我一直运行按计划进行胶水工作。昨晚我的 Glue Job 在调用 bucket.objects.filter(Prefix=productionDirectory):

后由于 botocore.exceptions.NoCredentialsError: Unable to locate credentials 而失败

我的印象是这是没有定义凭证文件的结果，但 AWS Glue 总是毫无问题地提取凭证。我只是重新运行我的工作，一切都很完美。作为参考，我通过以下方式定义了我的 Glue 客户端：glue = boto3.client('glue')。有没有人经历过这个？这只是一个边缘案例吗？

完整日志：

Traceback (most recent call last):
  File "/tmp/data-deployment", line 67, in <module>
    for obj in bucket.objects.filter(Prefix=productionDirectory):
  File "/home/spark/.local/lib/python3.7/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/home/spark/.local/lib/python3.7/site-packages/boto3/resources/collection.py", line 166, in pages
    for page in pages:
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/client.py", line 613, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/client.py", line 632, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/spark/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Edit/Update：这是一个已知错误。我在下面发布了 AWS 提供的缓解策略作为答案。

Answer 1

我遇到了与 Glue 类似的问题，但不完全相同。

我们将外部表与 SparkSQL 和 S3 一起使用，有时会突然出现异常，即 Table not found。该问题从未在测试中重现并且频率最低。由于我们的作业运行在重试时完全没问题，所以我们启用了重试机制来解决它。

它与 Glue 及其无服务器环境的内部运作有关。

Answer 2

更新：我通过支持联系了 AWS，他们做出了回应。显然这是一个已知的错误和问题。虽然他们没有解决方案或解决方案的预计到达时间，但他们确实有办法缓解这个问题。信息如下：

Thank you for reporting your issue to us and product team is aware of this intermittent issue. 
They are working on resolution however, I do not have an ETA. 
To mitigate this issue, increase the timeout / attempts to meta service request in your code:

####START######

import os

####Increase meta service timeout and attempt########

os.environ['AWS_METADATA_SERVICE_NUM_ATTEMPTS'] ="5"
os.environ['AWS_METADATA_SERVICE_TIMEOUT'] ="30"

#####################END#################

Glue Boto 客户端——NoCredentialsError

Glue Boto Client -- NoCredentialsError

amazon-web-services

boto3

aws-glue