如何在裸机上使用 hostNetwork 安装 nginx-ingress？
How to install nginx-ingress with hostNetwork on bare-metal?
我在裸机上创建了一个单节点 kubeadm 集群，经过一些研究后，我将采用主机网络方法 (https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network)，因为由于网络限制，NodePort 不是一个选项。
我尝试通过以下命令使用 helm chart 安装 nginx-ingress：
helm install stable/nginx-ingress \
--set controller.hostNetwork=true
问题是它正在创建一个永远挂起的 LoadBalancer 服务，并且我的入口对象没有被路由：
NAME READY STATUS RESTARTS AGE
pod/whopping-kitten-nginx-ingress-controller-5db858b48c-dp2j8 1/1 Running 0 5m34s
pod/whopping-kitten-nginx-ingress-default-backend-5c574f4449-dr4xm 1/1 Running 0 5m34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6m43s
service/whopping-kitten-nginx-ingress-controller LoadBalancer 10.97.143.40 <pending> 80:30068/TCP,443:30663/TCP 5m34s
service/whopping-kitten-nginx-ingress-default-backend ClusterIP 10.106.217.96 <none> 80/TCP 5m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/whopping-kitten-nginx-ingress-controller 1/1 1 1 5m34s
deployment.apps/whopping-kitten-nginx-ingress-default-backend 1/1 1 1 5m34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/whopping-kitten-nginx-ingress-controller-5db858b48c 1 1 1 5m34s
replicaset.apps/whopping-kitten-nginx-ingress-default-backend-5c574f4449 1 1 1 5m34s
是否需要进行任何其他配置才能成功使用此方法？
更新：这里是 ingress-controller pod 的日志
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.24.1
Build: git-ce418168f
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
I0707 19:02:50.552631 6 flags.go:185] Watching for Ingress class: nginx
W0707 19:02:50.552882 6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0707 19:02:50.556215 6 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0707 19:02:50.556368 6 main.go:205] Creating API client for https://10.96.0.1:443
I0707 19:02:50.562296 6 main.go:249] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0707 19:02:51.357524 6 main.go:102] Validated default/precise-bunny-nginx-ingress-default-backend as the default backend.
I0707 19:02:51.832384 6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0707 19:02:53.516654 6 store.go:613] Unexpected error reading configuration configmap: configmaps "precise-bunny-nginx-ingress-controller" not found
I0707 19:02:53.527297 6 nginx.go:265] Starting NGINX Ingress controller
I0707 19:02:54.630002 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"801", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/staging-ingress
I0707 19:02:54.727989 6 nginx.go:311] Starting NGINX process
I0707 19:02:54.728249 6 leaderelection.go:217] attempting to acquire leader lease default/ingress-controller-leader-nginx...
W0707 19:02:54.729235 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:54.729334 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:54.729442 6 controller.go:797] Service "default/test" does not have any active Endpoint.
I0707 19:02:54.729535 6 controller.go:170] Configuration changes detected, backend reload required.
I0707 19:02:54.891620 6 controller.go:188] Backend successfully reloaded.
I0707 19:02:54.891654 6 controller.go:202] Initial sync, sleeping for 1 second.
I0707 19:02:54.948639 6 leaderelection.go:227] successfully acquired lease default/ingress-controller-leader-nginx
I0707 19:02:54.949148 6 status.go:86] new leader elected: precise-bunny-nginx-ingress-controller-679b9557ff-n57mc
[07/Jul/2019:19:02:55 +0000]TCP200000.000
W0707 19:02:58.062645 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:58.062676 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:58.062686 6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:02.406151 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:03:02.406188 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:02.406357 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:02 +0000]TCP200000.000
W0707 19:03:05.739438 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:05.739467 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:05 +0000]TCP200000.001
W0707 19:03:09.072793 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:09.072820 6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:12.406121 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:12.406143 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:15 +0000]TCP200000.000
I0707 19:03:54.959607 6 status.go:295] updating Ingress default/staging-ingress status from [] to [{ }]
I0707 19:03:54.961925 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"1033", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/staging-ingress
我会说您需要某种接近于 Ingress on Custom Kubernetes 文章中提供的配置的配置。
并同意@Ijaz Ahmad Khan - 它应该是 kind: DaemonSet
示例：
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
hostNetwork: true
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
securityContext:
runAsNonRoot: false
@ijaz-ahmad-khan @vkr 给出了解决问题的好主意，但完整的设置步骤是：
1) 使用
安装 nginx-ingress
helm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet
2) 在您的部署中放置：
spec:
template:
spec:
hostNetwork: true
3) 在所有 Ingress 对象中放置：
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
在某些教程中，您可能会被建议简单地做一个
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml
但是，如果您在没有负载平衡器的私有 LAN 上使用裸机实现，则需要稍微修改此步骤。
不要直接应用 deploy.yaml，而是执行 wget（例如 wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml），然后编辑 deploy.yaml - 向下滚动到 Deployment 资源并添加 hostNetwork: true key:value 对。例如：
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
hostNetwork: true
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
等等
所以您正在做的是修改控制器部署模板的规范。
使用如下内容部署修改后的版本：
kubectl apply -f deploy.yaml
您为使用此 deployment/controller 而创建的 Ingress 资源应该这样引用它：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-api-ingress
annotations:
kubernetes.io/ingress.class: nginx
然后你可以这样描述：
kubectl describe service -n ingress-nginx ingress-nginx-controller
并找出哪个幸运节点被指定为您的入口。
我在裸机上创建了一个单节点 kubeadm 集群，经过一些研究后，我将采用主机网络方法 (https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network)，因为由于网络限制，NodePort 不是一个选项。
我尝试通过以下命令使用 helm chart 安装 nginx-ingress：
helm install stable/nginx-ingress \
--set controller.hostNetwork=true
问题是它正在创建一个永远挂起的 LoadBalancer 服务，并且我的入口对象没有被路由：
NAME READY STATUS RESTARTS AGE
pod/whopping-kitten-nginx-ingress-controller-5db858b48c-dp2j8 1/1 Running 0 5m34s
pod/whopping-kitten-nginx-ingress-default-backend-5c574f4449-dr4xm 1/1 Running 0 5m34s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6m43s
service/whopping-kitten-nginx-ingress-controller LoadBalancer 10.97.143.40 <pending> 80:30068/TCP,443:30663/TCP 5m34s
service/whopping-kitten-nginx-ingress-default-backend ClusterIP 10.106.217.96 <none> 80/TCP 5m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/whopping-kitten-nginx-ingress-controller 1/1 1 1 5m34s
deployment.apps/whopping-kitten-nginx-ingress-default-backend 1/1 1 1 5m34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/whopping-kitten-nginx-ingress-controller-5db858b48c 1 1 1 5m34s
replicaset.apps/whopping-kitten-nginx-ingress-default-backend-5c574f4449 1 1 1 5m34s
是否需要进行任何其他配置才能成功使用此方法？
更新：这里是 ingress-controller pod 的日志
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.24.1
Build: git-ce418168f
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
I0707 19:02:50.552631 6 flags.go:185] Watching for Ingress class: nginx
W0707 19:02:50.552882 6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0707 19:02:50.556215 6 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0707 19:02:50.556368 6 main.go:205] Creating API client for https://10.96.0.1:443
I0707 19:02:50.562296 6 main.go:249] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0707 19:02:51.357524 6 main.go:102] Validated default/precise-bunny-nginx-ingress-default-backend as the default backend.
I0707 19:02:51.832384 6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0707 19:02:53.516654 6 store.go:613] Unexpected error reading configuration configmap: configmaps "precise-bunny-nginx-ingress-controller" not found
I0707 19:02:53.527297 6 nginx.go:265] Starting NGINX Ingress controller
I0707 19:02:54.630002 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"801", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/staging-ingress
I0707 19:02:54.727989 6 nginx.go:311] Starting NGINX process
I0707 19:02:54.728249 6 leaderelection.go:217] attempting to acquire leader lease default/ingress-controller-leader-nginx...
W0707 19:02:54.729235 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:54.729334 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:54.729442 6 controller.go:797] Service "default/test" does not have any active Endpoint.
I0707 19:02:54.729535 6 controller.go:170] Configuration changes detected, backend reload required.
I0707 19:02:54.891620 6 controller.go:188] Backend successfully reloaded.
I0707 19:02:54.891654 6 controller.go:202] Initial sync, sleeping for 1 second.
I0707 19:02:54.948639 6 leaderelection.go:227] successfully acquired lease default/ingress-controller-leader-nginx
I0707 19:02:54.949148 6 status.go:86] new leader elected: precise-bunny-nginx-ingress-controller-679b9557ff-n57mc
[07/Jul/2019:19:02:55 +0000]TCP200000.000
W0707 19:02:58.062645 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:58.062676 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:58.062686 6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:02.406151 6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:03:02.406188 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:02.406357 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:02 +0000]TCP200000.000
W0707 19:03:05.739438 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:05.739467 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:05 +0000]TCP200000.001
W0707 19:03:09.072793 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:09.072820 6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:12.406121 6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:12.406143 6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:15 +0000]TCP200000.000
I0707 19:03:54.959607 6 status.go:295] updating Ingress default/staging-ingress status from [] to [{ }]
I0707 19:03:54.961925 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"1033", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/staging-ingress
我会说您需要某种接近于 Ingress on Custom Kubernetes 文章中提供的配置的配置。
并同意@Ijaz Ahmad Khan - 它应该是 kind: DaemonSet
示例：
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
hostNetwork: true
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
hostPort: 80
- name: https
containerPort: 443
hostPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
securityContext:
runAsNonRoot: false
@ijaz-ahmad-khan @vkr 给出了解决问题的好主意，但完整的设置步骤是：
1) 使用
安装 nginx-ingresshelm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet
2) 在您的部署中放置：
spec:
template:
spec:
hostNetwork: true
3) 在所有 Ingress 对象中放置：
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
在某些教程中，您可能会被建议简单地做一个
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml
但是，如果您在没有负载平衡器的私有 LAN 上使用裸机实现，则需要稍微修改此步骤。
不要直接应用 deploy.yaml，而是执行 wget（例如 wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml），然后编辑 deploy.yaml - 向下滚动到 Deployment 资源并添加 hostNetwork: true key:value 对。例如：
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
hostNetwork: true
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
等等
所以您正在做的是修改控制器部署模板的规范。
使用如下内容部署修改后的版本：
kubectl apply -f deploy.yaml
您为使用此 deployment/controller 而创建的 Ingress 资源应该这样引用它：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-api-ingress
annotations:
kubernetes.io/ingress.class: nginx
然后你可以这样描述：
kubectl describe service -n ingress-nginx ingress-nginx-controller
并找出哪个幸运节点被指定为您的入口。