我正在尝试将 externalname 与 Nginx-Controller 的 nodeport 服务一起使用，但我收到 502 Bad Gateway
I am trying to use externalname together with nodeport service for Nginx-Controller but I am getting 502 Bad Gateway
环境：
我有：
1- NGINX Ingress 控制器版本： 1.15.9，图片：0.23.0
2- Kubernetes 版本：
Client Version: version.Info{Major:"1", Minor:"13",
GitVersion:"v1.13.4",
GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1",
GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z",
GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13",
GitVersion:"v1.13.4",
GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1",
GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z",
GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
云提供商或硬件配置： KVM 上的虚拟机
OS（例如来自 /etc/os-release）：
NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel
fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
内核（例如 uname -a）：
Linux node01 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC
2019 x86_64 x86_64 x86_64 GNU/Linux
安装工具： kubeadm
更多详情：
CNI：编织
设置：
- 2 个弹性 HA 代理、3 个主节点、2 个基础节点和工作节点。
- 我将所有服务公开为节点端口，HA-Proxy 将它们重新分配给 public 虚拟 IP。
- 托管在承载监控和日志记录工具（Grafana、Prometheus、EFK 等）的基础设施节点上的专用项目
- 后端 NFS 存储作为持久存储
发生了什么：
我希望能够使用外部名称而不是节点端口，所以我不想通过 vip + 3000 访问 grafana，而是想通过 http://grafana.wild-card-dns-zone
访问它
部署
- 我创建了一个名为 ingress 的新命名空间
- 我部署如下：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: **2**
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
name: nginx-ingress
spec:
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
node-role.kubernetes.io/infra: infra
terminationGracePeriodSeconds: 60
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
args:
- /nginx-ingress-controller
- --default-backend-service=ingress/ingress-controller-nginx-ingress-default-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
- --v3
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
generation: 1
labels:
app: nginx-ingress
chart: nginx-ingress-1.3.1
component: default-backend
name: ingress-controller-nginx-ingress-default-backend
namespace: ingress
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
component: default-backend
release: ingress-controller
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
component: default-backend
release: ingress-controller
spec:
nodeSelector:
node-role.kubernetes.io/infra: infra
containers:
- image: k8s.gcr.io/defaultbackend:1.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: nginx-ingress-default-backend
ports:
- containerPort: 8080
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx-ingress
chart: nginx-ingress-1.3.1
component: default-backend
name: ingress-controller-nginx-ingress-default-backend
namespace: ingress
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: nginx-ingress
component: default-backend
release: ingress-controller
sessionAffinity: None
type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrolebinding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-rolebinding
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress
入口设置：
服务
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-03-25T16:03:01Z"
labels:
app: jaeger
app.kubernetes.io/component: query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jeager-query
namespace: monitoring-logging
resourceVersion: "3055947"
selfLink: /api/v1/namespaces/monitoring-logging/services/jeager-query
uid: 778550f0-4f17-11e9-9078-001a4a16021e
spec:
externalName: jaeger.example.com
ports:
- port: 16686
protocol: TCP
targetPort: 16686
selector:
app: jaeger
app.kubernetes.io/component: query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-03-25T15:40:30Z"
labels:
app: grafana
chart: grafana-2.2.4
heritage: Tiller
release: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3053698"
selfLink: /api/v1/namespaces/monitoring-logging/services/grafana
uid: 51b9d878-4f14-11e9-9078-001a4a16021e
spec:
externalName: grafana.example.com
ports:
- name: http
port: 3000
protocol: TCP
targetPort: 3000
selector:
app: grafana
release: grafana
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
入口
入口 1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/service-upstream: "true"
creationTimestamp: "2019-03-25T21:13:56Z"
generation: 1
labels:
app: jaeger
app.kubernetes.io/component: query-ingress
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jaeger-query
namespace: monitoring-logging
resourceVersion: "3111683"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/jaeger-query
uid: e6347f6b-4f42-11e9-9e8e-001a4a16021c
spec:
rules:
- host: jaeger.example.com
http:
paths:
- backend:
serviceName: jeager-query
servicePort: 16686
status:
loadBalancer: {}
入口 2
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace":"monitoring-logging"},"spec":{"rules":[{"host":"grafana.example.com","http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000}}]}}]}}
creationTimestamp: "2019-03-25T17:52:40Z"
generation: 1
labels:
app: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3071719"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/grafana
uid: c89d7f34-4f26-11e9-8c10-001a4a16021d
spec:
rules:
- host: grafana.example.com
http:
paths:
- backend:
serviceName: grafana
servicePort: 3000
status:
loadBalancer: {}
端点
端点 1
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2019-03-25T15:40:30Z"
labels:
app: grafana
chart: grafana-2.2.4
heritage: Tiller
release: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3050562"
selfLink: /api/v1/namespaces/monitoring-logging/endpoints/grafana
uid: 51bb1f9c-4f14-11e9-9e8e-001a4a16021c
subsets:
- addresses:
- ip: 10.42.0.15
nodeName: kuinfra01.example.com
targetRef:
kind: Pod
name: grafana-b44b4f867-bcq2x
namespace: monitoring-logging
resourceVersion: "1386975"
uid: 433e3d21-4827-11e9-9e8e-001a4a16021c
ports:
- name: http
port: 3000
protocol: TCP
端点 2
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2019-03-25T16:03:01Z"
labels:
app: jaeger
app.kubernetes.io/component: service-query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jeager-query
namespace: monitoring-logging
resourceVersion: "3114702"
selfLink: /api/v1/namespaces/monitoring-logging/endpoints/jeager-query
uid: 7786d833-4f17-11e9-9e8e-001a4a16021c
subsets:
- addresses:
- ip: 10.35.0.3
nodeName: kunode02.example.com
targetRef:
kind: Pod
name: jeager-query-7d9775d8f7-2hwdn
namespace: monitoring-logging
resourceVersion: "3114693"
uid: fdac9771-4f49-11e9-9e8e-001a4a16021c
ports:
- name: query
port: 16686
protocol: TCP
我能够从 ingress-controller pod 内部卷曲端点：
# kubectl exec -it nginx-ingress-controller-5dd67f88cc-z2g8s -n ingress -- /bin/bash
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl -k https://localhost
<a href="/login">Found</a>.
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl http://localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.15.9</center>
</body>
</html>
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ exit
但是当我试图到达 jaeger.example.com 或 grafana.example.com 时，我收到 502 错误网关和以下错误日志：
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /search HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 514 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 b7c813286fccf27fffa03eb6564edfd1
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://jeager.example.com/search" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 494 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 9e582912614e67dfee6be1f679de5933
I0325 16:40:32.497868 8 socket.go:225] skiping metric for host jeager.example.com that is not being served
I0325 16:40:32.497886 8 socket.go:225] skiping metric for host jeager.example.com that is not being served
首先感谢 cookiedough 提供有关服务问题的线索，但后来我遇到了使用外部名称创建服务的问题，但我发现我的错误感谢 "Long" 用户在 slack 中，错误是我使用的是 ExternalName 类型的服务，它应该是集群 IP 类型这里是解决问题的步骤（备注 https 问题是一个单独的问题）：
1- 创建指向 public IP 的通配符 DNS 区域
1- 对于新服务，只需创建 ClusterIP 类型的服务
2- 在服务的命名空间中使用以下示例 (yaml) 创建入口：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: grafana
name: grafana
namespace: grafana-namespace
spec:
rules:
host: grafana.example.com
http:
paths:
backend:
serviceName: grafana
servicePort: 3000
3- kubectl -f apply -f grafana-ingress.yaml
现在您可以在 http://grafana.example,com
访问您的 grafana
环境：
我有：
1- NGINX Ingress 控制器版本： 1.15.9，图片：0.23.0
2- Kubernetes 版本：
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
云提供商或硬件配置： KVM 上的虚拟机
OS（例如来自 /etc/os-release）：
NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
内核（例如 uname -a）：
Linux node01 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
安装工具： kubeadm
更多详情：
CNI：编织
设置：
- 2 个弹性 HA 代理、3 个主节点、2 个基础节点和工作节点。
- 我将所有服务公开为节点端口，HA-Proxy 将它们重新分配给 public 虚拟 IP。
- 托管在承载监控和日志记录工具（Grafana、Prometheus、EFK 等）的基础设施节点上的专用项目
- 后端 NFS 存储作为持久存储
发生了什么： 我希望能够使用外部名称而不是节点端口，所以我不想通过 vip + 3000 访问 grafana，而是想通过 http://grafana.wild-card-dns-zone
访问它部署
- 我创建了一个名为 ingress 的新命名空间
- 我部署如下：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: **2**
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
name: nginx-ingress
spec:
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
node-role.kubernetes.io/infra: infra
terminationGracePeriodSeconds: 60
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
args:
- /nginx-ingress-controller
- --default-backend-service=ingress/ingress-controller-nginx-ingress-default-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
- --v3
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
generation: 1
labels:
app: nginx-ingress
chart: nginx-ingress-1.3.1
component: default-backend
name: ingress-controller-nginx-ingress-default-backend
namespace: ingress
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
component: default-backend
release: ingress-controller
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
component: default-backend
release: ingress-controller
spec:
nodeSelector:
node-role.kubernetes.io/infra: infra
containers:
- image: k8s.gcr.io/defaultbackend:1.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: nginx-ingress-default-backend
ports:
- containerPort: 8080
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 60
---
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
name: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx-ingress
chart: nginx-ingress-1.3.1
component: default-backend
name: ingress-controller-nginx-ingress-default-backend
namespace: ingress
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: nginx-ingress
component: default-backend
release: ingress-controller
sessionAffinity: None
type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrolebinding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-rolebinding
namespace: ingress
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress
入口设置：
服务
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-03-25T16:03:01Z"
labels:
app: jaeger
app.kubernetes.io/component: query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jeager-query
namespace: monitoring-logging
resourceVersion: "3055947"
selfLink: /api/v1/namespaces/monitoring-logging/services/jeager-query
uid: 778550f0-4f17-11e9-9078-001a4a16021e
spec:
externalName: jaeger.example.com
ports:
- port: 16686
protocol: TCP
targetPort: 16686
selector:
app: jaeger
app.kubernetes.io/component: query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-03-25T15:40:30Z"
labels:
app: grafana
chart: grafana-2.2.4
heritage: Tiller
release: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3053698"
selfLink: /api/v1/namespaces/monitoring-logging/services/grafana
uid: 51b9d878-4f14-11e9-9078-001a4a16021e
spec:
externalName: grafana.example.com
ports:
- name: http
port: 3000
protocol: TCP
targetPort: 3000
selector:
app: grafana
release: grafana
sessionAffinity: None
type: ExternalName
status:
loadBalancer: {}
入口
入口 1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/service-upstream: "true"
creationTimestamp: "2019-03-25T21:13:56Z"
generation: 1
labels:
app: jaeger
app.kubernetes.io/component: query-ingress
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jaeger-query
namespace: monitoring-logging
resourceVersion: "3111683"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/jaeger-query
uid: e6347f6b-4f42-11e9-9e8e-001a4a16021c
spec:
rules:
- host: jaeger.example.com
http:
paths:
- backend:
serviceName: jeager-query
servicePort: 16686
status:
loadBalancer: {}
入口 2
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana","namespace":"monitoring-logging"},"spec":{"rules":[{"host":"grafana.example.com","http":{"paths":[{"backend":{"serviceName":"grafana","servicePort":3000}}]}}]}}
creationTimestamp: "2019-03-25T17:52:40Z"
generation: 1
labels:
app: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3071719"
selfLink: /apis/extensions/v1beta1/namespaces/monitoring-logging/ingresses/grafana
uid: c89d7f34-4f26-11e9-8c10-001a4a16021d
spec:
rules:
- host: grafana.example.com
http:
paths:
- backend:
serviceName: grafana
servicePort: 3000
status:
loadBalancer: {}
端点
端点 1
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2019-03-25T15:40:30Z"
labels:
app: grafana
chart: grafana-2.2.4
heritage: Tiller
release: grafana
name: grafana
namespace: monitoring-logging
resourceVersion: "3050562"
selfLink: /api/v1/namespaces/monitoring-logging/endpoints/grafana
uid: 51bb1f9c-4f14-11e9-9e8e-001a4a16021c
subsets:
- addresses:
- ip: 10.42.0.15
nodeName: kuinfra01.example.com
targetRef:
kind: Pod
name: grafana-b44b4f867-bcq2x
namespace: monitoring-logging
resourceVersion: "1386975"
uid: 433e3d21-4827-11e9-9e8e-001a4a16021c
ports:
- name: http
port: 3000
protocol: TCP
端点 2
apiVersion: v1
kind: Endpoints
metadata:
creationTimestamp: "2019-03-25T16:03:01Z"
labels:
app: jaeger
app.kubernetes.io/component: service-query
app.kubernetes.io/instance: jeager
app.kubernetes.io/managed-by: jaeger-operator
app.kubernetes.io/name: jeager-query
app.kubernetes.io/part-of: jaeger
name: jeager-query
namespace: monitoring-logging
resourceVersion: "3114702"
selfLink: /api/v1/namespaces/monitoring-logging/endpoints/jeager-query
uid: 7786d833-4f17-11e9-9e8e-001a4a16021c
subsets:
- addresses:
- ip: 10.35.0.3
nodeName: kunode02.example.com
targetRef:
kind: Pod
name: jeager-query-7d9775d8f7-2hwdn
namespace: monitoring-logging
resourceVersion: "3114693"
uid: fdac9771-4f49-11e9-9e8e-001a4a16021c
ports:
- name: query
port: 16686
protocol: TCP
我能够从 ingress-controller pod 内部卷曲端点：
# kubectl exec -it nginx-ingress-controller-5dd67f88cc-z2g8s -n ingress -- /bin/bash
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl -k https://localhost
<a href="/login">Found</a>.
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ curl http://localhost
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.15.9</center>
</body>
</html>
www-data@nginx-ingress-controller-5dd67f88cc-z2g8s:/etc/nginx$ exit
但是当我试图到达 jaeger.example.com 或 grafana.example.com 时，我收到 502 错误网关和以下错误日志：
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /search HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 514 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 b7c813286fccf27fffa03eb6564edfd1
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
2019/03/25 16:40:32 [error] 2816#2816: *4617326 connect() failed (111: Connection refused) while connecting to upstream, client: 10.39.0.0, server: _, request: "GET /favicon.ico HTTP/1.1", upstream: "http://vip:16686/favicon.ico", host: "jeager.example.com", referrer: "http://jeager.example.com/search"
10.39.0.0 - [10.39.0.0] - - [25/Mar/2019:16:40:32 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://jeager.example.com/search" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" 494 0.001 [monitoring-logging-jeager-query-16686] vip:16686, vip:16686, vip:16686 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 9e582912614e67dfee6be1f679de5933
I0325 16:40:32.497868 8 socket.go:225] skiping metric for host jeager.example.com that is not being served
I0325 16:40:32.497886 8 socket.go:225] skiping metric for host jeager.example.com that is not being served
首先感谢 cookiedough 提供有关服务问题的线索，但后来我遇到了使用外部名称创建服务的问题，但我发现我的错误感谢 "Long" 用户在 slack 中，错误是我使用的是 ExternalName 类型的服务，它应该是集群 IP 类型这里是解决问题的步骤（备注 https 问题是一个单独的问题）： 1- 创建指向 public IP 的通配符 DNS 区域 1- 对于新服务，只需创建 ClusterIP 类型的服务 2- 在服务的命名空间中使用以下示例 (yaml) 创建入口：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: grafana
name: grafana
namespace: grafana-namespace
spec:
rules:
host: grafana.example.com
http:
paths:
backend:
serviceName: grafana
servicePort: 3000
3- kubectl -f apply -f grafana-ingress.yaml 现在您可以在 http://grafana.example,com
访问您的 grafana