ScrapydWeb：连接在 docker-compose 内被拒绝

Question

我尝试运行几个 scrapyd 服务在我的本地主机上有一个简单的集群，但只有第一个节点工作。对于其他 2 个，我收到以下错误

scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_3', port=6802): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060b8ef50>: Failed to establish a new connection: [Errno 111] Connection refused'))
scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_2', port=6801): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060a1e650>: Failed to establish a new connection: [Errno 111] Connection refused'))

我有以下 docker-compose.yml 文件：

version: '3'
services:
  scrapyd_node_1:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6800:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_2:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6801:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_3:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6802:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped


  scrapydweb:
    build:
      context: .
      dockerfile: ./crawlers/scrapydweb/Dockerfile
    environment:
      USERNAME: "test"
      PASSWORD: "test"
      SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"
    links:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3
    ports:
      - "5000:5000"
    depends_on:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3

    restart: unless-stopped

我的 docker-compose 文件有什么问题？

Answer 1

问题所在行：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"

尝试将其更改为：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6800,scrapyd_node_3:6800"

解释：

例如，当您定义 docker 服务 scrapyd_node_2 时，您将端口定义为：

    ports:
      - "6801:6800"

这意味着，contanier 的端口 6800 映射到主机上的端口 6801。因此，当你想用主机名 scrapyd_node_2 声明节点时，你应该使用它的端口 = scrapyd_node_2:6800.

ScrapydWeb：连接在 docker-compose 内被拒绝

ScrapydWeb: Connection refused within docker-compose

python

scrapy

scrapyd

docker

docker-compose