将 Google Cloud KMS 与 Flask、gunicorn 和 gevent worker 一起使用时出错:"google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded"
Error using Google Cloud KMS with Flask, gunicorn and gevent worker: "google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded"
在 Flask 请求的上下文中尝试使用 google-cloud-kms 进行加密或解密会导致在使用 gunicorn 和 gevent worker 提供 Flask 应用程序时出现以下错误。当 运行 本地应用程序和 Google App Engine(灵活)环境中时都会发生错误。
app_1 | Traceback (most recent call last):
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
app_1 | return callable_(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
app_1 | return _end_unary_response_blocking(state, call, False, None)
app_1 | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
app_1 | raise _Rendezvous(state, None, None, deadline)
app_1 | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
app_1 | status = StatusCode.UNAVAILABLE
app_1 | details = "Deadline Exceeded"
app_1 | debug_error_string = "{"created":"@1561236057.820157100","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
app_1 | >
app_1 |
app_1 | The above exception was the direct cause of the following exception:
app_1 |
app_1 | Traceback (most recent call last):
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
app_1 | response = self.full_dispatch_request()
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
app_1 | rv = self.handle_user_exception(e)
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
app_1 | reraise(exc_type, exc_value, tb)
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
app_1 | raise value
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
app_1 | rv = self.dispatch_request()
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
app_1 | return self.view_functions[rule.endpoint](**req.view_args)
app_1 | File "/app/app.py", line 14, in encrypt
app_1 | encrypted = client.encrypt(key_name, plaintext.encode())
app_1 | File "/usr/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py", line 1286, in encrypt
app_1 | request, retry=retry, timeout=timeout, metadata=metadata
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
app_1 | return wrapped_func(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
app_1 | on_error=on_error,
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
app_1 | return target()
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
app_1 | return func(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
app_1 | six.raise_from(exceptions.from_grpc_error(exc), exc)
app_1 | File "<string>", line 3, in raise_from
app_1 | google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
requirements.txt
Flask==1.0.3
gevent==1.4.0
google-cloud-kms==1.0.0
grpcio==1.21.1
gunicorn==19.9.0
重现此内容的 Flask 应用示例:
from flask import Flask
from google.cloud import kms_v1
import base64
def create_app():
app = Flask(__name__)
@app.route('/')
def encrypt():
plaintext = "plain"
client = kms_v1.KeyManagementServiceClient()
key_name = client.crypto_key_path('project', 'location', 'keyring', 'key')
encrypted = client.encrypt(key_name, plaintext.encode())
return base64.urlsafe_b64encode(encrypted.ciphertext)
return app
example_app = create_app()
搭配:
gunicorn -t 300 -b :8080 -k gevent -w 4 app:example_app
回答我自己的问题,因为我花了一些时间调试它,并在网上找到了类似但不是这个特定问题的答案。
问题似乎是 grpc 和 gevent 如何一起工作(或不工作),最快的解决方法是 使用另一个 gunicorn worker。经过测试,至少 sync、eventlet 和 gthread worker 可以防止发生此错误。
gunicorn -t 300 -b :8080 -k eventlet -w 4 app:example_app
在 Flask 请求的上下文中尝试使用 google-cloud-kms 进行加密或解密会导致在使用 gunicorn 和 gevent worker 提供 Flask 应用程序时出现以下错误。当 运行 本地应用程序和 Google App Engine(灵活)环境中时都会发生错误。
app_1 | Traceback (most recent call last):
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
app_1 | return callable_(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
app_1 | return _end_unary_response_blocking(state, call, False, None)
app_1 | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
app_1 | raise _Rendezvous(state, None, None, deadline)
app_1 | grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
app_1 | status = StatusCode.UNAVAILABLE
app_1 | details = "Deadline Exceeded"
app_1 | debug_error_string = "{"created":"@1561236057.820157100","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":14}"
app_1 | >
app_1 |
app_1 | The above exception was the direct cause of the following exception:
app_1 |
app_1 | Traceback (most recent call last):
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
app_1 | response = self.full_dispatch_request()
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
app_1 | rv = self.handle_user_exception(e)
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
app_1 | reraise(exc_type, exc_value, tb)
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
app_1 | raise value
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
app_1 | rv = self.dispatch_request()
app_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
app_1 | return self.view_functions[rule.endpoint](**req.view_args)
app_1 | File "/app/app.py", line 14, in encrypt
app_1 | encrypted = client.encrypt(key_name, plaintext.encode())
app_1 | File "/usr/local/lib/python3.7/site-packages/google/cloud/kms_v1/gapic/key_management_service_client.py", line 1286, in encrypt
app_1 | request, retry=retry, timeout=timeout, metadata=metadata
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
app_1 | return wrapped_func(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
app_1 | on_error=on_error,
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 182, in retry_target
app_1 | return target()
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
app_1 | return func(*args, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
app_1 | six.raise_from(exceptions.from_grpc_error(exc), exc)
app_1 | File "<string>", line 3, in raise_from
app_1 | google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
requirements.txt
Flask==1.0.3
gevent==1.4.0
google-cloud-kms==1.0.0
grpcio==1.21.1
gunicorn==19.9.0
重现此内容的 Flask 应用示例:
from flask import Flask
from google.cloud import kms_v1
import base64
def create_app():
app = Flask(__name__)
@app.route('/')
def encrypt():
plaintext = "plain"
client = kms_v1.KeyManagementServiceClient()
key_name = client.crypto_key_path('project', 'location', 'keyring', 'key')
encrypted = client.encrypt(key_name, plaintext.encode())
return base64.urlsafe_b64encode(encrypted.ciphertext)
return app
example_app = create_app()
搭配:
gunicorn -t 300 -b :8080 -k gevent -w 4 app:example_app
回答我自己的问题,因为我花了一些时间调试它,并在网上找到了类似但不是这个特定问题的答案。
问题似乎是 grpc 和 gevent 如何一起工作(或不工作),最快的解决方法是 使用另一个 gunicorn worker。经过测试,至少 sync、eventlet 和 gthread worker 可以防止发生此错误。
gunicorn -t 300 -b :8080 -k eventlet -w 4 app:example_app