ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback raised exception! (gunicorn) when using eventlet (async) workers
ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback raised exception! (gunicorn) when using eventlet (async) workers
我编写了一个运行良好的 Flask 应用程序 - 完全符合我的要求，当通过 flask run 使用 Flask 的开发服务器时。这是一个漫长的 web-scraping 过程，使用了大量 Google 云库。
部署到 Google App Engine 后，我发现我必须用 gunicorn 包装我的 Flask api。好的，没问题，我在本地安装了它，运行 和以前一样。但是突然间，现在，我收到了一个全新的错误，我不知道如何调试 - 这是堆栈跟踪：
[2020-07-20 05:26:45 -0400] [7354] [INFO] Starting gunicorn 20.0.4
[2020-07-20 05:26:45 -0400] [7354] [INFO] Listening at: http://127.0.0.1:8000 (7354)
[2020-07-20 05:26:45 -0400] [7354] [INFO] Using worker: eventlet
[2020-07-20 05:26:45 -0400] [7356] [INFO] Booting worker with pid: 7356
WARNING:root:course-collect manually triggered
ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fc3f7710970>" raised exception!
Traceback (most recent call last):
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/grpc/_plugin_wrapping.py", line 77, in __call__
self._metadata_plugin(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 84, in __call__
callback(self._get_authorization_headers(context), None)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 70, in _get_authorization_headers
self._credentials.before_request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request
self.refresh(request)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/service_account.py", line 359, in refresh
access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
response_data = _token_endpoint_request(request, token_uri, body)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request
response = request(method="POST", url=token_uri, headers=headers, body=body)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/requests.py", line 180, in __call__
response = self.session.request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connection.py", line 342, in connect
self.ssl_context = create_urllib3_context(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 276, in create_urllib3_context
context.options |= options
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 476 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
malloc(): mismatching next->prev_size (unsorted)
[2020-07-20 05:27:29 -0400] [7361] [INFO] Booting worker with pid: 7361
我目前 运行 我的应用使用命令 gunicorn --worker-class eventlet app:app（使用 app.py 和 app = Flask(__name__)）。
当我只切换到 gunicorn app:app gunicorn 时，其运行与 Flask 相同。
但问题是，知道这个 api 的端点需要“未定义的时间量”，进行“阻塞调用”，否则 requests 会访问很多网页，我的应用程序似乎是使用异步 (eventlet/gevent) worker 的千篇一律的情况。
Choosing a Worker Type
The default synchronous workers assume that your application is resource-bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. An example of something that takes an undefined amount of time is a request to the internet. At some point the external network will fail in such a way that clients will pile up on your servers. So, in this sense, any web application which makes outgoing requests to APIs will benefit from an asynchronous worker.
This resource bound assumption is why we require a buffering proxy in front of a default configuration Gunicorn. If you exposed synchronous workers to the internet, a DOS attack would be trivial by creating a load that trickles data to the servers. For the curious, Hey is an example of this type of load.
Some examples of behavior requiring asynchronous workers:
- Applications making long blocking calls (Ie, external web services)
- Serving requests directly to the internet
- Streaming requests and responses
- Long polling
- Web sockets
- Comet
有人可以指出异步工作程序破坏我的应用程序的原因（如果它有点明显）（粘贴在下面）
app.py
from flask import Flask
import logging
from firebase_admin import firestore, _apps, initialize_app, credentials
from google.cloud.storage import Client
from google.cloud.scheduler_v1 import CloudSchedulerClient
from google.api_core.exceptions import NotFound, GoogleAPICallError, PermissionDenied
app = Flask(__name__)
@app.route('/init')
def start_process():
start_time = time()
storage_client = Client()
scheduler_client = CloudSchedulerClient()
scheduler_path = scheduler_client.location_path(config.PROJECT_ID, config.REGION_ID)
cred = credentials.ApplicationDefault()
try:
scheduler_client.delete_job(f"{scheduler_path}/jobs/{config.CRON_NAME}")
except GoogleAPICallError or PermissionDenied:
logging.warning("course-collect manually triggered")
# I had more code here but even all commented out, this error still happened
return "200 OK"
gRPC 不适用于 eventlet。但是 gRPC 在其实验性 API 中确实有一个 gevent 模式。有两种选择：
- 切换到仅 HTTP 传输，某些 GCP Python 客户端确实有此模式；
- 切换到 gevent worker 查看 code。
我编写了一个运行良好的 Flask 应用程序 - 完全符合我的要求，当通过 flask run 使用 Flask 的开发服务器时。这是一个漫长的 web-scraping 过程，使用了大量 Google 云库。
部署到 Google App Engine 后，我发现我必须用 gunicorn 包装我的 Flask api。好的，没问题，我在本地安装了它，运行 和以前一样。但是突然间，现在，我收到了一个全新的错误，我不知道如何调试 - 这是堆栈跟踪：
[2020-07-20 05:26:45 -0400] [7354] [INFO] Starting gunicorn 20.0.4
[2020-07-20 05:26:45 -0400] [7354] [INFO] Listening at: http://127.0.0.1:8000 (7354)
[2020-07-20 05:26:45 -0400] [7354] [INFO] Using worker: eventlet
[2020-07-20 05:26:45 -0400] [7356] [INFO] Booting worker with pid: 7356
WARNING:root:course-collect manually triggered
ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fc3f7710970>" raised exception!
Traceback (most recent call last):
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/grpc/_plugin_wrapping.py", line 77, in __call__
self._metadata_plugin(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 84, in __call__
callback(self._get_authorization_headers(context), None)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/grpc.py", line 70, in _get_authorization_headers
self._credentials.before_request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request
self.refresh(request)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/service_account.py", line 359, in refresh
access_token, expiry, _ = _client.jwt_grant(request, self._token_uri, assertion)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 153, in jwt_grant
response_data = _token_endpoint_request(request, token_uri, body)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request
response = request(method="POST", url=token_uri, headers=headers, body=body)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/google/auth/transport/requests.py", line 180, in __call__
response = self.session.request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/connection.py", line 342, in connect
self.ssl_context = create_urllib3_context(
File "/mnt/c/Users/*******/Projects/course_collect/venv/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 276, in create_urllib3_context
context.options |= options
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.8/ssl.py", line 602, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 476 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
malloc(): mismatching next->prev_size (unsorted)
[2020-07-20 05:27:29 -0400] [7361] [INFO] Booting worker with pid: 7361
我目前 运行 我的应用使用命令 gunicorn --worker-class eventlet app:app（使用 app.py 和 app = Flask(__name__)）。
当我只切换到 gunicorn app:app gunicorn 时，其运行与 Flask 相同。
但问题是，知道这个 api 的端点需要“未定义的时间量”，进行“阻塞调用”，否则 requests 会访问很多网页，我的应用程序似乎是使用异步 (eventlet/gevent) worker 的千篇一律的情况。
Choosing a Worker Type
The default synchronous workers assume that your application is resource-bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. An example of something that takes an undefined amount of time is a request to the internet. At some point the external network will fail in such a way that clients will pile up on your servers. So, in this sense, any web application which makes outgoing requests to APIs will benefit from an asynchronous worker.
This resource bound assumption is why we require a buffering proxy in front of a default configuration Gunicorn. If you exposed synchronous workers to the internet, a DOS attack would be trivial by creating a load that trickles data to the servers. For the curious, Hey is an example of this type of load.
Some examples of behavior requiring asynchronous workers:
- Applications making long blocking calls (Ie, external web services)
- Serving requests directly to the internet
- Streaming requests and responses
- Long polling
- Web sockets
- Comet
有人可以指出异步工作程序破坏我的应用程序的原因（如果它有点明显）（粘贴在下面）
app.py
from flask import Flask
import logging
from firebase_admin import firestore, _apps, initialize_app, credentials
from google.cloud.storage import Client
from google.cloud.scheduler_v1 import CloudSchedulerClient
from google.api_core.exceptions import NotFound, GoogleAPICallError, PermissionDenied
app = Flask(__name__)
@app.route('/init')
def start_process():
start_time = time()
storage_client = Client()
scheduler_client = CloudSchedulerClient()
scheduler_path = scheduler_client.location_path(config.PROJECT_ID, config.REGION_ID)
cred = credentials.ApplicationDefault()
try:
scheduler_client.delete_job(f"{scheduler_path}/jobs/{config.CRON_NAME}")
except GoogleAPICallError or PermissionDenied:
logging.warning("course-collect manually triggered")
# I had more code here but even all commented out, this error still happened
return "200 OK"
gRPC 不适用于 eventlet。但是 gRPC 在其实验性 API 中确实有一个 gevent 模式。有两种选择：
- 切换到仅 HTTP 传输，某些 GCP Python 客户端确实有此模式；
- 切换到 gevent worker 查看 code。