uWSGI lazy-apps 和 ThreadPool

Question

运行 uWSGI 与

$ cat /etc/uwsgi/uwsgi.cfg

[uwsgi]
callable = app
socket = /var/run/arivale-service/uwsgi.sock
chmod-socket = 666
pidfile = /var/run/arivale-service/uwsgi.pid
master = true
enable-threads = true
single-interpreter = true
thunder-lock
need-app
processes = 4

未启用 lazy-apps，对调用以下端点的请求挂起

import boto3
# ...irrelevant imports
from multiprocessing.dummy import Pool as ThreadPool


POOL = ThreadPool(6)

# ...irrelevant setup


def get_ecs_task_definitions(service_name):
    ecs_service_name, _ = get_ecs_service_name_and_cluster(service_name)

    def get_task_definition(task_definition_arn):
        formatted_task_definition = {}
        task_definition = ECS.describe_task_definition(taskDefinition=task_definition_arn)['taskDefinition']
        # ...
        # build formatted_task_definition from task_definition
        # ...
        return formatted_task_definition
    task_definition_arns = ECS.list_task_definitions(
        familyPrefix=ecs_service_name, status='ACTIVE')['taskDefinitionArns']
    return POOL.map(get_task_definition, task_definition_arns)


@service.api('/api/services/<service_name>/ecs/task-definitions')
def get_task_definitions(service_name):
    return {'service_name': service_name, 'task_definitions': get_ecs_task_definitions(service_name)}

NGINX 正在平衡 uWSGI 应用程序，这就是它在 error.log (NGINX) 中所说的：

Jun 10 03:54:26 93e04e04e2cf nginx_error: 2017/06/10 03:54:26 [error] 49#49: *33 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.16.254.95, server: localhost, request: "GET /api/services/data-analysis-service/ecs/task-definitions HTTP/1.1",upstream: "uwsgi://unix:/var/run/arivale-service/uwsgi.sock", host: "devops-service.arivale.com"

对端点的每个请求都会挂起一个worker（下面是uwsgitop两次请求后的输出）：

uwsgi-2.0.15 - Sat Jun 10 21:26:10 2017 - req: 0 - RPS: 0 - lq: 0 - tx: 0
node: localhost - cwd: /var/www/arivale-service/web - uid: 0 - gid: 0 - masterpid: 45
 WID    %       PID     REQ     RPS     EXC     SIG     STATUS  AVG     RSS     VSZ     TX      ReSpwn  HC      RunT    LastSpwn
 1      0.0     74      0       0       0       0       busy    0ms     0       0       0       1       0       0       21:23:20
 2      0.0     75      0       0       0       0       busy    0ms     0       0       0       1       0       0       21:23:20
 3      0.0     76      0       0       0       0       idle    0ms     0       0       0       1       0       0       21:23:20
 4      0.0     77      0       0       0       0       idle    0ms     0       0       0       1       0       0       21:23:20

启用lazy-apps 修复了这个问题。有谁确切知道为什么？

Answer 1

发生这种情况是因为 uWSGI worker 无法访问在 master 中创建的线程池，参见。

使用 @postfork 解决这个问题：

global THREAD_POOL = None

@postfork
def _make_thread_pool():
    global THREAD_POOL
    THREAD_POOL = ThreadPool(8)

uWSGI lazy-apps 和 ThreadPool

uWSGI lazy-apps and ThreadPool

multithreading

nginx

threadpool

uwsgi