尝试通过 Consul Connect Sidecar 代理连接到 mongodb 服务
Trying to connect to mongodb service through Consul Connect Sidecar Proxy
我设置了一个 Minikube 并在其中设置了一个 mongo 实例 运行ning。我使用 Consul + Consul Connect 来构建我的服务。只有我无法使用 sidecar upstreams 从另一个服务连接到 mongo，发生了一些奇怪的事情...
我的 mongo 实例是使用 bitnami helm chart 安装的，我只是设置服务名称，设置用户名并更改存储 class 以满足我的需要，并将服务网格的 consul 注释放在pod 注释部分：
image:
registry: docker.io
repository: bitnami/mongodb
tag: 4.2.5-debian-10-r3
pullPolicy: IfNotPresent
debug: false
serviceAccount:
create: true
name: "svc-identity-data"
usePassword: true
mongodbRootPassword: rootpassword
mongodbUsername: identity
mongodbPassword: identity
mongodbDatabase: company
service:
name: svc-identity-data
annotations: {}
type: ClusterIP
port: 27017
useStatefulSet: true
replicaSet:
enabled: false
useHostnames: true
name: rs0
replicas:
secondary: 1
arbiter: 1
pdb:
enabled: true
minAvailable:
primary: 1
secondary: 1
arbiter: 1
annotations: {}
labels: {}
podAnnotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service": "svc-identity-data"
"consul.hashicorp.com/connect-service-protocol": "tcp"
persistence:
enabled: true
mountPath: /bitnami/mongodb
subPath: ""
storageClass: "standard"
accessModes:
- ReadWriteOnce
size: 8Gi
annotations: {}
configmap:
storage:
dbPath: /bitnami/mongodb/data/db
journal:
enabled: true
directoryPerDB: false
systemLog:
destination: file
quiet: false
logAppend: true
logRotate: reopen
path: /opt/bitnami/mongodb/logs/mongodb.log
verbosity: 0
net:
port: 27017
unixDomainSocket:
enabled: true
pathPrefix: /opt/bitnami/mongodb/tmp
ipv6: false
bindIp: 0.0.0.0
processManagement:
fork: false
pidFilePath: /opt/bitnami/mongodb/tmp/mongodb.pid
setParameter:
enableLocalhostAuthBypass: true
security:
authorization: enabled
其次，我启动了一个独立的 mongodb pod 来使用 mongo 客户端，并使用注解与 consul connect 建立了网格
apiVersion: v1
kind: Pod
metadata:
name: mongo-client
labels:
name: mongo-client
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service-upstreams": "svc-identity-data:28017"
"consul.hashicorp.com/connect-service-protocol": "tcp"
spec:
containers:
- name: mongo-client
image: mongo:4.2.5
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 27017
我现在有一个 mongodb 服务和一个 mongo 客户端 pod，上游到 mongodb 服务绑定在 127.0.0.1:28017
当我尝试使用我的上游连接到 mongodb 服务时，我遇到了一个我不理解的行为
> kubectl exec -it mongo-client mongo --host 127.0.0.1 --port 28017 -u root -p rootpassword
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("8c46012d-8083-4029-8495-167bbe8bf063") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
这里没问题，一切对我来说都很好，但是如果我使用 mongo 和连接字符串而不是单独的参数，我会得到一个连接被拒绝的
> kubectl exec -it mongo-client mongo mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-22T15:04:07.955+0000 I NETWORK [js] DBClientConnection failed to receive message from 127.0.0.1:28017 - HostUnreachable: Connection closed by peer
2020-04-22T15:04:07.968+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host '127.0.0.1:28017' :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-22T15:04:07.973+0000 F - [main] exception: connect failed
2020-04-22T15:04:07.973+0000 E - [main] exiting with code 1
我完全不明白使用连接字符串和单独参数有什么区别，如果您有任何线索或解决方案，请告诉我。
P.S : 我没有设置任何安全通信 (tls)，我在 minikube 上（因为我是微服务架构和 Kubernetes n00b），它是为了试验服务网格（我们需要生活在当前时代），涉及不使用 sidecar 连接到服务的解决方案不是重点，顺便说一句，使用连接字符串直接连接到服务可以完美地工作。
> kubectl exec -it mongo-client mongo -mongodb://root:roopassword@svc-identity-data:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://svc-identity-data:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("713febaf-2000-4ca6-8b1f-963c76986e72") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
EDIT ：重启 minikube 使一切按预期工作。我将对此事进行更多调查以了解原因。也许其他人会遇到同样的问题。
编辑 2：我发现一件事：通过 sidecar 连接到 mongo 时出现连接错误是随机的，当我 运行 命令成功时，这是我得到的
root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-24T12:51:15.641+0000 I NETWORK [js] DBClientConnection failed to receive message from localhost:28017 - HostUnreachable: Connection closed by peer
2020-04-24T12:51:15.702+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host 'localhost:28017' :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-24T12:51:15.729+0000 F - [main] exception: connect failed
2020-04-24T12:51:15.729+0000 E - [main] exiting with code 1
root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("628bfcf9-6d44-4168-ab74-19a717d746f6") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten]
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
并且在 mongo 日志的一侧：
2020-04-24T12:51:19.281+0000 I NETWORK [conn6647] end connection 127.0.0.1:54148 (6 connections now open)
2020-04-24T12:51:19.526+0000 I COMMAND [conn6646] command admin.$cmd appName: "MongoDB Shell" command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-256", payload: "xxx", $db: "admin" } numYields:0 reslen:196 locks:{} protocol:op_msg 231ms
2020-04-24T12:51:19.938+0000 I ACCESS [conn6646] Successfully authenticated as principal root on admin from client 127.0.0.1:54142
2020-04-24T12:51:20.024+0000 I NETWORK [listener] connection accepted from 127.0.0.1:54168 #6648 (7 connections now open)
2020-04-24T12:51:20.027+0000 I NETWORK [conn6648] received client metadata from 127.0.0.1:54168 conn6648: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.5" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 10 (buster)"", architecture: "x86_64", version: "Kernel 4.19.94" } }
2020-04-24T12:51:20.215+0000 I NETWORK [conn6648] end connection 127.0.0.1:54168 (6 connections now open)
2020-04-24T12:51:21.328+0000 I NETWORK [conn6646] end connection 127.0.0.1:54142 (5 connections now open)
我越来越糊涂了，我无法解释那个行为。
问题可能是证书的 CN 与 MongoDB 配置文件中主机名的值不匹配。它是关于MongoDB规格和参数的，你运行它。
证书的 CN（通用名称）或 SAN（主题备用名称）必须与您在 运行 mongo 时提供的 --hostname 的值相匹配。
您的 MongoDB URI 是：
MONGODB_URI=mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin
MongoDB 不在本地主机上。此外 MongoDB 服务器需要允许任何主机连接到数据库。默认情况下，它只允许来自相同运行时的连接。您需要使用数据库容器获取分配给 pod 的服务的 IP 地址 - svc-identity-data 的地址为 10.107.99.51.
我找到了解决方案，事实证明这是最简单的问题：资源
我的 minikube 不足以让所有 pods 运行 迅速完成，它在 sidecar 代理 pods 之间引入了延迟，即使 kubenetes 在任何中断时都没有引发错误。
我是一名 kubernetes 学习者，所以我没有立即想到它。既然我知道发生了什么，我可以朝着正确的方向进行调查，以了解延长延迟的原因可能是一个问题。
我设置了一个 Minikube 并在其中设置了一个 mongo 实例 运行ning。我使用 Consul + Consul Connect 来构建我的服务。只有我无法使用 sidecar upstreams 从另一个服务连接到 mongo，发生了一些奇怪的事情...
我的 mongo 实例是使用 bitnami helm chart 安装的，我只是设置服务名称，设置用户名并更改存储 class 以满足我的需要，并将服务网格的 consul 注释放在pod 注释部分：
image:
registry: docker.io
repository: bitnami/mongodb
tag: 4.2.5-debian-10-r3
pullPolicy: IfNotPresent
debug: false
serviceAccount:
create: true
name: "svc-identity-data"
usePassword: true
mongodbRootPassword: rootpassword
mongodbUsername: identity
mongodbPassword: identity
mongodbDatabase: company
service:
name: svc-identity-data
annotations: {}
type: ClusterIP
port: 27017
useStatefulSet: true
replicaSet:
enabled: false
useHostnames: true
name: rs0
replicas:
secondary: 1
arbiter: 1
pdb:
enabled: true
minAvailable:
primary: 1
secondary: 1
arbiter: 1
annotations: {}
labels: {}
podAnnotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service": "svc-identity-data"
"consul.hashicorp.com/connect-service-protocol": "tcp"
persistence:
enabled: true
mountPath: /bitnami/mongodb
subPath: ""
storageClass: "standard"
accessModes:
- ReadWriteOnce
size: 8Gi
annotations: {}
configmap:
storage:
dbPath: /bitnami/mongodb/data/db
journal:
enabled: true
directoryPerDB: false
systemLog:
destination: file
quiet: false
logAppend: true
logRotate: reopen
path: /opt/bitnami/mongodb/logs/mongodb.log
verbosity: 0
net:
port: 27017
unixDomainSocket:
enabled: true
pathPrefix: /opt/bitnami/mongodb/tmp
ipv6: false
bindIp: 0.0.0.0
processManagement:
fork: false
pidFilePath: /opt/bitnami/mongodb/tmp/mongodb.pid
setParameter:
enableLocalhostAuthBypass: true
security:
authorization: enabled
其次，我启动了一个独立的 mongodb pod 来使用 mongo 客户端，并使用注解与 consul connect 建立了网格
apiVersion: v1
kind: Pod
metadata:
name: mongo-client
labels:
name: mongo-client
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service-upstreams": "svc-identity-data:28017"
"consul.hashicorp.com/connect-service-protocol": "tcp"
spec:
containers:
- name: mongo-client
image: mongo:4.2.5
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 27017
我现在有一个 mongodb 服务和一个 mongo 客户端 pod，上游到 mongodb 服务绑定在 127.0.0.1:28017
当我尝试使用我的上游连接到 mongodb 服务时，我遇到了一个我不理解的行为
> kubectl exec -it mongo-client mongo --host 127.0.0.1 --port 28017 -u root -p rootpassword
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("8c46012d-8083-4029-8495-167bbe8bf063") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
这里没问题，一切对我来说都很好，但是如果我使用 mongo 和连接字符串而不是单独的参数，我会得到一个连接被拒绝的
> kubectl exec -it mongo-client mongo mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-22T15:04:07.955+0000 I NETWORK [js] DBClientConnection failed to receive message from 127.0.0.1:28017 - HostUnreachable: Connection closed by peer
2020-04-22T15:04:07.968+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host '127.0.0.1:28017' :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-22T15:04:07.973+0000 F - [main] exception: connect failed
2020-04-22T15:04:07.973+0000 E - [main] exiting with code 1
我完全不明白使用连接字符串和单独参数有什么区别，如果您有任何线索或解决方案，请告诉我。
P.S : 我没有设置任何安全通信 (tls)，我在 minikube 上（因为我是微服务架构和 Kubernetes n00b），它是为了试验服务网格（我们需要生活在当前时代），涉及不使用 sidecar 连接到服务的解决方案不是重点，顺便说一句，使用连接字符串直接连接到服务可以完美地工作。
> kubectl exec -it mongo-client mongo -mongodb://root:roopassword@svc-identity-data:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://svc-identity-data:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("713febaf-2000-4ca6-8b1f-963c76986e72") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten]
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
EDIT ：重启 minikube 使一切按预期工作。我将对此事进行更多调查以了解原因。也许其他人会遇到同样的问题。
编辑 2：我发现一件事：通过 sidecar 连接到 mongo 时出现连接错误是随机的，当我 运行 命令成功时，这是我得到的
root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-24T12:51:15.641+0000 I NETWORK [js] DBClientConnection failed to receive message from localhost:28017 - HostUnreachable: Connection closed by peer
2020-04-24T12:51:15.702+0000 E QUERY [js] Error: network error while attempting to run command 'isMaster' on host 'localhost:28017' :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-24T12:51:15.729+0000 F - [main] exception: connect failed
2020-04-24T12:51:15.729+0000 E - [main] exiting with code 1
root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("628bfcf9-6d44-4168-ab74-19a717d746f6") }
MongoDB server version: 4.2.5
Server has startup warnings:
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten]
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-24T06:43:39.359+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>
bye
并且在 mongo 日志的一侧：
2020-04-24T12:51:19.281+0000 I NETWORK [conn6647] end connection 127.0.0.1:54148 (6 connections now open)
2020-04-24T12:51:19.526+0000 I COMMAND [conn6646] command admin.$cmd appName: "MongoDB Shell" command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-256", payload: "xxx", $db: "admin" } numYields:0 reslen:196 locks:{} protocol:op_msg 231ms
2020-04-24T12:51:19.938+0000 I ACCESS [conn6646] Successfully authenticated as principal root on admin from client 127.0.0.1:54142
2020-04-24T12:51:20.024+0000 I NETWORK [listener] connection accepted from 127.0.0.1:54168 #6648 (7 connections now open)
2020-04-24T12:51:20.027+0000 I NETWORK [conn6648] received client metadata from 127.0.0.1:54168 conn6648: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.5" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 10 (buster)"", architecture: "x86_64", version: "Kernel 4.19.94" } }
2020-04-24T12:51:20.215+0000 I NETWORK [conn6648] end connection 127.0.0.1:54168 (6 connections now open)
2020-04-24T12:51:21.328+0000 I NETWORK [conn6646] end connection 127.0.0.1:54142 (5 connections now open)
我越来越糊涂了，我无法解释那个行为。
问题可能是证书的 CN 与 MongoDB 配置文件中主机名的值不匹配。它是关于MongoDB规格和参数的，你运行它。
证书的 CN（通用名称）或 SAN（主题备用名称）必须与您在 运行 mongo 时提供的 --hostname 的值相匹配。
您的 MongoDB URI 是：
MONGODB_URI=mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin
MongoDB 不在本地主机上。此外 MongoDB 服务器需要允许任何主机连接到数据库。默认情况下，它只允许来自相同运行时的连接。您需要使用数据库容器获取分配给 pod 的服务的 IP 地址 - svc-identity-data 的地址为 10.107.99.51.
我找到了解决方案，事实证明这是最简单的问题：资源
我的 minikube 不足以让所有 pods 运行 迅速完成，它在 sidecar 代理 pods 之间引入了延迟，即使 kubenetes 在任何中断时都没有引发错误。
我是一名 kubernetes 学习者，所以我没有立即想到它。既然我知道发生了什么，我可以朝着正确的方向进行调查，以了解延长延迟的原因可能是一个问题。