docker-compose 中的分布式安全 MinIO

Question

我在 Docker 中有一个相当复杂的系统。所有运行都在一个大 docker-compose 文件中。以前我的 Docker Swarm 中的一个（管理器）节点上的所有内容运行所以我为我的域生成了一个 CERT（certbot）并且我使用了下面的 MinIO我的撰写文件中的服务：

  object_storage:
    image: minio/minio:RELEASE.2020-12-10T01-54-29Z
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
    command: server /data
    depends_on:
      - fluentd
    volumes:
      - object_storage_data:/data
      - ./certs/domain.crt:/root/.minio/certs/public.crt
      - ./certs/domain.key:/root/.minio/certs/private.key
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

以上实现按预期工作！但现在我有 2 个独立的服务器运行 MinIO。这些服务器作为工作节点加入到我的 Docker Swarm 中。 MinIO 不应在管理器节点上运行（仅在两个独立的工作节点上）！

>>> docker node ls
ID                                          HOSTNAME    STATUS      AVAILABILITY   MANAGER STATUS   ENGINE VERSION
mcbkz9m5nzf7oa3fiqk0lf4qo *  manager         Ready           Active                    Leader                    20.10.1
dz4e3k70g8ik2z4bcx8u0ft9ao   minio_1          Ready           Active                                                   20.10.2
r0qpdn2guyy5773vo8vg2trzo    minio_2          Ready           Active                                                   20.10.2

我的 docker-compose 文件中的当前 MinIO 实现：

object_storage_1:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000:9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_1_1:/data1
     - object_storage_data_1_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_1
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

 object_storage_2:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_2_1:/data1
     - object_storage_data_2_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_2
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

如果我检查我的 MinIO 服务实例的日志，我收到以下错误：

Unable to read 'format.json' from https://object_storage_1:9000/data1: Post "https://object_storage_1:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_1
Unable to read 'format.json' from https://object_storage_2:9000/data1: Post "https://object_storage_2:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_2

但是我可以在9000端口上到达MinIO，只是弹出错误：

我只想通过我的域 (my_domain.app:9000) 访问 MinIO。 MinIO 在这种情况下不使用真实服务器名称，而是使用“虚拟”Docker 网络（例如：https://object_storage_2:9000）。

我的问题：

如何为“虚拟”Docker 网络（例如：object_storage_1 或 object_storage_2）生成证书？
我应该把生成的证书放在哪里？
是否可以仅使用我生成的（为我的域）证书来解决？

我愿意接受每一个提示和解决方案！

Answer 1

我不得不将（域）CERT 文件放入 minio/certs/CAs 文件夹而不是 /root/.minio/certs 文件夹。此外，我必须将 CERT 复制到工作节点（独立的服务器），如果我没有将它复制到服务在工作节点上找不到它的节点。

正确的 volumes 参数如下所示：

volumes:
  - object_storage_data_1_1:/data1
  - object_storage_data_1_2:/data2
  - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt

我的几个 MinIO 服务中的一个工作服务：

  object-storage-1:
    image: minio/minio:RELEASE.2021-08-17T20-53-08Z
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
      MINIO_BROWSER_REDIRECT_URL: https://${SYSTEM_HOST}:9001
      MINIO_SERVER_URL: https://${SYSTEM_HOST}:9000
    command: server --console-address ":9001" http://object-storage-{1...4}/data{1...2}
    hostname: object-storage-1
    depends_on:
      - fluentd
    volumes:
      - object_storage_data_1_1:/data1
      - object_storage_data_1_2:/data2
      - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == minio1
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

并且我必须创建一个 NgInx 配置：

upstream minio {
    server object-storage-1:9000;
    server object-storage-2:9000;
    server object-storage-3:9000;
    server object-storage-4:9000;
}

upstream console {
    ip_hash;
    server object-storage-1:9001;
    server object-storage-2:9001;
    server object-storage-3:9001;
    server object-storage-4:9001;
}

server {
    listen              9000 ssl;
    listen              [::]:9000 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;


    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio;

    }
}

server {
    listen              9001 ssl;
    listen              [::]:9001 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://console;

    }
}

docker-compose 中的分布式安全 MinIO

Distributed secure MinIO in docker-compose

docker

docker-compose

certificate

minio

object-storage