Increase of requests count end up with 'Error: socket hang up'

Question

后端 service/nginx 代理在请求数量增加时开始响应 'Error: socket hang up'。设置如下。

OS: 分OS 6

Express JS 服务 -> nginx 作为代理 -> flask 应用程序 运行 by Gunicorn

JS 应用程序同时向其他服务发送多个请求，当请求计数超过 ~100 时，它开始 return 错误响应。如果计数较低，一切正常。

我遵循了 Gunicorn 文档中的 nginx 配置示例 + 增加超时限制 + 增加 nginx 打开文件限制。我也尝试过 keepalive 选项，但问题仍然存在。 Gunicorn 没有显示任何错误。

nginx 配置片段：

upstream app_server {
    server 127.0.0.1:8000 fail_timeout=0;
    keepalive 100;
}

server {
    listen 5001;
    client_max_body_size 4G;

    keepalive_timeout 300;

    root /path/to/app/current/public; # static files

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        # Timeouts
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        send_timeout 300;

        proxy_http_version 1.1;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://app_server;
    }
}

从代理收到错误响应：

{ RequestError: Error: socket hang up
    at new RequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:185:22)
    at Request.emit (events.js:160:13)
    at Request.onRequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:881:8)
    at ClientRequest.emit (events.js:160:13)
    at Socket.socketOnEnd (_http_client.js:423:9)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)
  name: 'RequestError',
  message: 'Error: socket hang up',
  cause: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  error: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  options:
   { method: 'PUT',
     uri: 'http://localhost:5001/transformers/segmentAvg',
     qs:
      { stdMultiplier: 2,
        segmentLeft: 1509366682333,
        segmentRight: 1509367401685 },
     body: { index: [Array], values: [Array] },
     headers: {},
     json: true,
     callback: [Function: RP$callback],
     transform: undefined,
     simple: true,
     resolveWithFullResponse: false,
     transform2xxOnly: false },
  response: undefined }

已添加：

在 OS 日志中记录了以下条目：

possible SYN flooding on port X. Sending cookies.

Answer 1

内核套接字积压已达到限制并丢弃了以下请求。

原因：由于 Red Hat Enterprise 中的 LISTEN 套接字缓冲区已满，内核丢弃 TCP 连接 Linux

增加内核套接字积压限制

检查当前值：

# sysctl net.core.somaxconn
net.core.somaxconn = 128

增加数值：

# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048

再次查看以确认更改：

# sysctl net.core.somaxconn
net.core.somaxconn = 2048

坚持更改：

echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf

增加应用程序套接字侦听积压

uWSGI的配置参数

listen=1024

---

此解决方案取自 https://access.redhat.com/solutions/30453