kong-ingress-controller 的 EXTERNAL_IP 待定
kong-ingress-controller's EXTERNAL_IP is pending
我已经在 3 节点 k8s 集群（裸机）上使用 yaml 文件安装了 kong-ingress-controller（您可以在问题底部看到该文件）并且一切正常并运行：
$kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default bar-deployment-64d5b5b5c4-99p4q 1/1 Running 0 12m
default foo-deployment-877bf9974-xmpj6 1/1 Running 0 15m
kong ingress-kong-5cd9db4db9-4cg4q 2/2 Running 0 79m
kube-system calico-kube-controllers-5f6cfd688c-5njnn 1/1 Running 0 18h
kube-system calico-node-5k9b6 1/1 Running 0 18h
kube-system calico-node-jbb7k 1/1 Running 0 18h
kube-system calico-node-mmmts 1/1 Running 0 18h
kube-system coredns-74ff55c5b-5q5fn 1/1 Running 0 23h
kube-system coredns-74ff55c5b-9bbbk 1/1 Running 0 23h
kube-system etcd-kubernetes-master 1/1 Running 1 23h
kube-system kube-apiserver-kubernetes-master 1/1 Running 1 23h
kube-system kube-controller-manager-kubernetes-master 1/1 Running 1 23h
kube-system kube-proxy-4h7hs 1/1 Running 0 20h
kube-system kube-proxy-sd6b2 1/1 Running 0 20h
kube-system kube-proxy-v9z8p 1/1 Running 1 23h
kube-system kube-scheduler-kubernetes-master 1/1 Running 1 23h
但问题就在这里:
EXTERNAL_IP kong-proxy service pending 所以我无法从外部访问我的集群
$kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default bar-service ClusterIP 10.103.49.102 <none> 5000/TCP 15m
default foo-service ClusterIP 10.102.52.89 <none> 5000/TCP 19m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
kong kong-proxy LoadBalancer 10.104.79.161 <pending> 80:31583/TCP,443:30053/TCP 82m
kong kong-validation-webhook ClusterIP 10.109.75.104 <none> 443/TCP 82m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23h
$ kubectl describe service kong-proxy -n kong
Name: kong-proxy
Namespace: kong
Labels: <none>
Annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector: app=ingress-kong
Type: LoadBalancer
IP Families: <none>
IP: 10.104.79.161
IPs: 10.104.79.161
Port: proxy 80/TCP
TargetPort: 8000/TCP
NodePort: proxy 31583/TCP
Endpoints: 192.168.74.69:8000
Port: proxy-ssl 443/TCP
TargetPort: 8443/TCP
NodePort: proxy-ssl 30053/TCP
Endpoints: 192.168.74.69:8443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
我的 k8s 版本是 1.20.1 并且
我的 docker 版本是 19.3.10 。
如果有人能帮助我找到解决方案，那就太棒了
=============================================
kong-ingress-controller yaml 文件：
apiVersion: v1
kind: Namespace
metadata:
name: kong
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongclusterplugins.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .plugin
description: Name of the plugin
name: Plugin-Type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .disabled
description: Indicates if the plugin is disabled
name: Disabled
priority: 1
type: boolean
- JSONPath: .config
description: Configuration of the plugin
name: Config
priority: 1
type: string
group: configuration.konghq.com
names:
kind: KongClusterPlugin
plural: kongclusterplugins
shortNames:
- kcp
scope: Cluster
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
config:
type: object
configFrom:
properties:
secretKeyRef:
properties:
key:
type: string
name:
type: string
namespace:
type: string
required:
- name
- namespace
- key
type: object
type: object
disabled:
type: boolean
plugin:
type: string
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
run_on:
enum:
- first
- second
- all
type: string
required:
- plugin
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongconsumers.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .username
description: Username of a Kong Consumer
name: Username
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
group: configuration.konghq.com
names:
kind: KongConsumer
plural: kongconsumers
shortNames:
- kc
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
credentials:
items:
type: string
type: array
custom_id:
type: string
username:
type: string
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongingresses.configuration.konghq.com
spec:
group: configuration.konghq.com
names:
kind: KongIngress
plural: kongingresses
shortNames:
- ki
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
proxy:
properties:
connect_timeout:
minimum: 0
type: integer
path:
pattern: ^/.*$
type: string
protocol:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
read_timeout:
minimum: 0
type: integer
retries:
minimum: 0
type: integer
write_timeout:
minimum: 0
type: integer
type: object
route:
properties:
headers:
additionalProperties:
items:
type: string
type: array
type: object
https_redirect_status_code:
type: integer
methods:
items:
type: string
type: array
path_handling:
enum:
- v0
- v1
type: string
preserve_host:
type: boolean
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
regex_priority:
type: integer
request_buffering:
type: boolean
response_buffering:
type: boolean
snis:
items:
type: string
type: array
strip_path:
type: boolean
upstream:
properties:
algorithm:
enum:
- round-robin
- consistent-hashing
- least-connections
type: string
hash_fallback:
type: string
hash_fallback_header:
type: string
hash_on:
type: string
hash_on_cookie:
type: string
hash_on_cookie_path:
type: string
hash_on_header:
type: string
healthchecks:
properties:
active:
properties:
concurrency:
minimum: 1
type: integer
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
http_path:
pattern: ^/.*$
type: string
timeout:
minimum: 0
type: integer
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
passive:
properties:
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
threshold:
type: integer
type: object
host_header:
type: string
slots:
minimum: 10
type: integer
type: object
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongplugins.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .plugin
description: Name of the plugin
name: Plugin-Type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .disabled
description: Indicates if the plugin is disabled
name: Disabled
priority: 1
type: boolean
- JSONPath: .config
description: Configuration of the plugin
name: Config
priority: 1
type: string
group: configuration.konghq.com
names:
kind: KongPlugin
plural: kongplugins
shortNames:
- kp
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
config:
type: object
configFrom:
properties:
secretKeyRef:
properties:
key:
type: string
name:
type: string
required:
- name
- key
type: object
type: object
disabled:
type: boolean
plugin:
type: string
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
run_on:
enum:
- first
- second
- all
type: string
required:
- plugin
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tcpingresses.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .status.loadBalancer.ingress[*].ip
description: Address of the load balancer
name: Address
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
group: configuration.konghq.com
names:
kind: TCPIngress
plural: tcpingresses
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
rules:
items:
properties:
backend:
properties:
serviceName:
type: string
servicePort:
format: int32
type: integer
type: object
host:
type: string
port:
format: int32
type: integer
type: object
type: array
tls:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
type: object
status:
type: object
version: v1beta1
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kong-serviceaccount
namespace: kong
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: kong-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
- extensions
- networking.internal.knative.dev
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
- extensions
- networking.internal.knative.dev
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- configuration.konghq.com
resources:
- tcpingresses/status
verbs:
- update
- apiGroups:
- configuration.konghq.com
resources:
- kongplugins
- kongclusterplugins
- kongcredentials
- kongconsumers
- kongingresses
- tcpingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kong-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kong-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: kong-serviceaccount
namespace: kong
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb
name: kong-proxy
namespace: kong
spec:
ports:
- name: proxy
port: 80
protocol: TCP
targetPort: 8000
- name: proxy-ssl
port: 443
protocol: TCP
targetPort: 8443
selector:
app: ingress-kong
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: kong-validation-webhook
namespace: kong
spec:
ports:
- name: webhook
port: 443
protocol: TCP
targetPort: 8080
selector:
app: ingress-kong
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ingress-kong
name: ingress-kong
namespace: kong
spec:
replicas: 1
selector:
matchLabels:
app: ingress-kong
template:
metadata:
annotations:
kuma.io/gateway: enabled
prometheus.io/port: "8100"
prometheus.io/scrape: "true"
traffic.sidecar.istio.io/includeInboundPorts: ""
labels:
app: ingress-kong
spec:
containers:
- env:
- name: KONG_PROXY_LISTEN
value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
- name: KONG_PORT_MAPS
value: 80:8000, 443:8443
- name: KONG_ADMIN_LISTEN
value: 127.0.0.1:8444 ssl
- name: KONG_STATUS_LISTEN
value: 0.0.0.0:8100
- name: KONG_DATABASE
value: "off"
- name: KONG_NGINX_WORKER_PROCESSES
value: "2"
- name: KONG_ADMIN_ACCESS_LOG
value: /dev/stdout
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
- name: KONG_PROXY_ERROR_LOG
value: /dev/stderr
image: kong:2.5
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- kong quit
livenessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 8100
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: proxy
ports:
- containerPort: 8000
name: proxy
protocol: TCP
- containerPort: 8443
name: proxy-ssl
protocol: TCP
- containerPort: 8100
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 8100
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
- env:
- name: CONTROLLER_KONG_ADMIN_URL
value: https://127.0.0.1:8444
- name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
value: "true"
- name: CONTROLLER_PUBLISH_SERVICE
value: kong/kong-proxy
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: kong/kubernetes-ingress-controller:1.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: ingress-controller
ports:
- containerPort: 8080
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
serviceAccountName: kong-serviceaccount
有同样的问题，在寻找解决方案几天后，我发现 metallb, from nginx ingress 在裸机上安装
MetalLB provides a network load-balancer implementation for Kubernetes
clusters that do not run on a supported cloud provider, effectively
allowing the usage of LoadBalancer Services within any cluster
，从他们的文档中我得到了这个
Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. The implementations of network load balancers that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.
我没有完成安装，但我希望上面的解释能回答您关于外部 ip 上的待处理状态的问题
简短的回答是@iglen_ 所说的 但我决定解释解决方案。
使用云提供商时，服务的 LoadBalancer 类型将由环境管理和提供（请参阅为此构建的 k8s docs) automatically, but when creating your own baremetal cluster you will need to add the service which will manage provisioning IPs for LoadBalancer type Services. One such service is Metal-LB。
安装 MetalLB 前检查 requirements.
在部署 MetalLB 之前，我们需要执行一个步骤：
If you’re using kube-proxy in IPVS mode, since Kubernetes v1.14.2 you
have to enable strict ARP mode.
请注意，如果您使用 kube-router 作为服务代理，则不需要它，因为它默认启用严格的 ARP。
输入此命令：
$ kubectl edit configmap -n kube-system kube-proxy
在打开的页面中搜索 mode，在我的例子中，mode 等于空字符串，所以我不需要更改任何东西，但在这种情况下如安装指南中所述设置为 ipvs 您需要在此文件中设置以下配置：
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
strictARP: true
作为下一步，您需要 运行 这些命令：
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
在 运行 上面的命令之后，我们有这个：
$ kubectl get all -n metallb-system
NAME READY STATUS RESTARTS AGE
pod/controller-6b78sff7d9-2rv2f 1/1 Running 0 3m
pod/speaker-7bqev 1/1 Running 0 3m
pod/speaker-txrg5 1/1 Running 0 3m
pod/speaker-w7th5 1/1 Running 0 3m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 3m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 3m
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-6b78sff7d9 1 1 1 3m
MetalLB 需要一些 IPv4 addresses :
$ ip a s
1: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
[...]
inet 10.240.1.59/24 brd 10.240.1.255 scope global dynamic noprefixroute ens160
valid_lft 425669sec preferred_lft 421269sec
[...]
ens160 是我的控制平面网络接口，如您所见，它的 IP 范围是 10.240.1.59/24 所以我要在这个网络中分配一组 IP 地址：
$ sipcalc 10.240.1.59/24
-[ipv4 : 10.240.1.59/24] - 0
[CIDR]
Host address - 10.240.1.59
Host address (decimal) - 183500115
Host address (hex) - AF0031B
Network address - 10.240.1.0
Network mask - 255.255.255.0
Network mask (bits) - 24
Network mask (hex) - FFFFF000
Broadcast address - 10.240.1.255
Cisco wildcard - 0.0.0.255
Addresses in network - 256
Network range - 10.240.1.0 - 10.240.1.255
Usable range - 10.240.1.1 - 10.240.1.254
现在我要从 Usable range 中获取 10 个 IP 地址并将其分配给 MetalLB。
让我们为 MetalLB 创建一个 configmap :
$ sudo nano metallb-cm.yaml
将以下配置粘贴到 metallb-cm.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.240.1.100-10.240.1.110
然后保存文件和运行这个命令：
$ kubectl create -f metallb-cm.yaml
现在让我们再次检查我们的服务：
$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default bar-service ClusterIP 10.103.49.102 <none> 5000/TCP 15m
default foo-service ClusterIP 10.102.52.89 <none> 5000/TCP 19m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
kong kong-proxy LoadBalancer 10.104.79.161 10.240.1.100 80:31583/TCP,443:30053/TCP 82m
kong kong-validation-webhook ClusterIP 10.109.75.104 <none> 443/TCP 82m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23h
如您所见，LoadBalancer 类型的服务现在有一个 IP 地址。
我已经在 3 节点 k8s 集群（裸机）上使用 yaml 文件安装了 kong-ingress-controller（您可以在问题底部看到该文件）并且一切正常并运行：
$kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default bar-deployment-64d5b5b5c4-99p4q 1/1 Running 0 12m
default foo-deployment-877bf9974-xmpj6 1/1 Running 0 15m
kong ingress-kong-5cd9db4db9-4cg4q 2/2 Running 0 79m
kube-system calico-kube-controllers-5f6cfd688c-5njnn 1/1 Running 0 18h
kube-system calico-node-5k9b6 1/1 Running 0 18h
kube-system calico-node-jbb7k 1/1 Running 0 18h
kube-system calico-node-mmmts 1/1 Running 0 18h
kube-system coredns-74ff55c5b-5q5fn 1/1 Running 0 23h
kube-system coredns-74ff55c5b-9bbbk 1/1 Running 0 23h
kube-system etcd-kubernetes-master 1/1 Running 1 23h
kube-system kube-apiserver-kubernetes-master 1/1 Running 1 23h
kube-system kube-controller-manager-kubernetes-master 1/1 Running 1 23h
kube-system kube-proxy-4h7hs 1/1 Running 0 20h
kube-system kube-proxy-sd6b2 1/1 Running 0 20h
kube-system kube-proxy-v9z8p 1/1 Running 1 23h
kube-system kube-scheduler-kubernetes-master 1/1 Running 1 23h
但问题就在这里:
EXTERNAL_IP kong-proxy service pending 所以我无法从外部访问我的集群
$kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default bar-service ClusterIP 10.103.49.102 <none> 5000/TCP 15m
default foo-service ClusterIP 10.102.52.89 <none> 5000/TCP 19m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
kong kong-proxy LoadBalancer 10.104.79.161 <pending> 80:31583/TCP,443:30053/TCP 82m
kong kong-validation-webhook ClusterIP 10.109.75.104 <none> 443/TCP 82m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23h
$ kubectl describe service kong-proxy -n kong
Name: kong-proxy
Namespace: kong
Labels: <none>
Annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector: app=ingress-kong
Type: LoadBalancer
IP Families: <none>
IP: 10.104.79.161
IPs: 10.104.79.161
Port: proxy 80/TCP
TargetPort: 8000/TCP
NodePort: proxy 31583/TCP
Endpoints: 192.168.74.69:8000
Port: proxy-ssl 443/TCP
TargetPort: 8443/TCP
NodePort: proxy-ssl 30053/TCP
Endpoints: 192.168.74.69:8443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
我的 k8s 版本是 1.20.1 并且 我的 docker 版本是 19.3.10 。 如果有人能帮助我找到解决方案，那就太棒了
=============================================
kong-ingress-controller yaml 文件：
apiVersion: v1
kind: Namespace
metadata:
name: kong
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongclusterplugins.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .plugin
description: Name of the plugin
name: Plugin-Type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .disabled
description: Indicates if the plugin is disabled
name: Disabled
priority: 1
type: boolean
- JSONPath: .config
description: Configuration of the plugin
name: Config
priority: 1
type: string
group: configuration.konghq.com
names:
kind: KongClusterPlugin
plural: kongclusterplugins
shortNames:
- kcp
scope: Cluster
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
config:
type: object
configFrom:
properties:
secretKeyRef:
properties:
key:
type: string
name:
type: string
namespace:
type: string
required:
- name
- namespace
- key
type: object
type: object
disabled:
type: boolean
plugin:
type: string
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
run_on:
enum:
- first
- second
- all
type: string
required:
- plugin
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongconsumers.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .username
description: Username of a Kong Consumer
name: Username
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
group: configuration.konghq.com
names:
kind: KongConsumer
plural: kongconsumers
shortNames:
- kc
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
credentials:
items:
type: string
type: array
custom_id:
type: string
username:
type: string
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongingresses.configuration.konghq.com
spec:
group: configuration.konghq.com
names:
kind: KongIngress
plural: kongingresses
shortNames:
- ki
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
proxy:
properties:
connect_timeout:
minimum: 0
type: integer
path:
pattern: ^/.*$
type: string
protocol:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
read_timeout:
minimum: 0
type: integer
retries:
minimum: 0
type: integer
write_timeout:
minimum: 0
type: integer
type: object
route:
properties:
headers:
additionalProperties:
items:
type: string
type: array
type: object
https_redirect_status_code:
type: integer
methods:
items:
type: string
type: array
path_handling:
enum:
- v0
- v1
type: string
preserve_host:
type: boolean
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
regex_priority:
type: integer
request_buffering:
type: boolean
response_buffering:
type: boolean
snis:
items:
type: string
type: array
strip_path:
type: boolean
upstream:
properties:
algorithm:
enum:
- round-robin
- consistent-hashing
- least-connections
type: string
hash_fallback:
type: string
hash_fallback_header:
type: string
hash_on:
type: string
hash_on_cookie:
type: string
hash_on_cookie_path:
type: string
hash_on_header:
type: string
healthchecks:
properties:
active:
properties:
concurrency:
minimum: 1
type: integer
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
http_path:
pattern: ^/.*$
type: string
timeout:
minimum: 0
type: integer
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
passive:
properties:
healthy:
properties:
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
successes:
minimum: 0
type: integer
type: object
unhealthy:
properties:
http_failures:
minimum: 0
type: integer
http_statuses:
items:
type: integer
type: array
interval:
minimum: 0
type: integer
tcp_failures:
minimum: 0
type: integer
timeout:
minimum: 0
type: integer
type: object
type: object
threshold:
type: integer
type: object
host_header:
type: string
slots:
minimum: 10
type: integer
type: object
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kongplugins.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .plugin
description: Name of the plugin
name: Plugin-Type
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
- JSONPath: .disabled
description: Indicates if the plugin is disabled
name: Disabled
priority: 1
type: boolean
- JSONPath: .config
description: Configuration of the plugin
name: Config
priority: 1
type: string
group: configuration.konghq.com
names:
kind: KongPlugin
plural: kongplugins
shortNames:
- kp
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
config:
type: object
configFrom:
properties:
secretKeyRef:
properties:
key:
type: string
name:
type: string
required:
- name
- key
type: object
type: object
disabled:
type: boolean
plugin:
type: string
protocols:
items:
enum:
- http
- https
- grpc
- grpcs
- tcp
- tls
type: string
type: array
run_on:
enum:
- first
- second
- all
type: string
required:
- plugin
version: v1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tcpingresses.configuration.konghq.com
spec:
additionalPrinterColumns:
- JSONPath: .status.loadBalancer.ingress[*].ip
description: Address of the load balancer
name: Address
type: string
- JSONPath: .metadata.creationTimestamp
description: Age
name: Age
type: date
group: configuration.konghq.com
names:
kind: TCPIngress
plural: tcpingresses
scope: Namespaced
subresources:
status: {}
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
rules:
items:
properties:
backend:
properties:
serviceName:
type: string
servicePort:
format: int32
type: integer
type: object
host:
type: string
port:
format: int32
type: integer
type: object
type: array
tls:
items:
properties:
hosts:
items:
type: string
type: array
secretName:
type: string
type: object
type: array
type: object
status:
type: object
version: v1beta1
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kong-serviceaccount
namespace: kong
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: kong-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
- extensions
- networking.internal.knative.dev
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
- extensions
- networking.internal.knative.dev
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- configuration.konghq.com
resources:
- tcpingresses/status
verbs:
- update
- apiGroups:
- configuration.konghq.com
resources:
- kongplugins
- kongclusterplugins
- kongcredentials
- kongconsumers
- kongingresses
- tcpingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kong-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kong-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: kong-serviceaccount
namespace: kong
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb
name: kong-proxy
namespace: kong
spec:
ports:
- name: proxy
port: 80
protocol: TCP
targetPort: 8000
- name: proxy-ssl
port: 443
protocol: TCP
targetPort: 8443
selector:
app: ingress-kong
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: kong-validation-webhook
namespace: kong
spec:
ports:
- name: webhook
port: 443
protocol: TCP
targetPort: 8080
selector:
app: ingress-kong
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: ingress-kong
name: ingress-kong
namespace: kong
spec:
replicas: 1
selector:
matchLabels:
app: ingress-kong
template:
metadata:
annotations:
kuma.io/gateway: enabled
prometheus.io/port: "8100"
prometheus.io/scrape: "true"
traffic.sidecar.istio.io/includeInboundPorts: ""
labels:
app: ingress-kong
spec:
containers:
- env:
- name: KONG_PROXY_LISTEN
value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
- name: KONG_PORT_MAPS
value: 80:8000, 443:8443
- name: KONG_ADMIN_LISTEN
value: 127.0.0.1:8444 ssl
- name: KONG_STATUS_LISTEN
value: 0.0.0.0:8100
- name: KONG_DATABASE
value: "off"
- name: KONG_NGINX_WORKER_PROCESSES
value: "2"
- name: KONG_ADMIN_ACCESS_LOG
value: /dev/stdout
- name: KONG_ADMIN_ERROR_LOG
value: /dev/stderr
- name: KONG_PROXY_ERROR_LOG
value: /dev/stderr
image: kong:2.5
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- kong quit
livenessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 8100
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: proxy
ports:
- containerPort: 8000
name: proxy
protocol: TCP
- containerPort: 8443
name: proxy-ssl
protocol: TCP
- containerPort: 8100
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /status
port: 8100
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
- env:
- name: CONTROLLER_KONG_ADMIN_URL
value: https://127.0.0.1:8444
- name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
value: "true"
- name: CONTROLLER_PUBLISH_SERVICE
value: kong/kong-proxy
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: kong/kubernetes-ingress-controller:1.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: ingress-controller
ports:
- containerPort: 8080
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
serviceAccountName: kong-serviceaccount
有同样的问题，在寻找解决方案几天后，我发现 metallb, from nginx ingress 在裸机上安装
MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster
，从他们的文档中我得到了这个
Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. The implementations of network load balancers that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.
我没有完成安装，但我希望上面的解释能回答您关于外部 ip 上的待处理状态的问题
简短的回答是@iglen_ 所说的
使用云提供商时，服务的 LoadBalancer 类型将由环境管理和提供（请参阅为此构建的 k8s docs) automatically, but when creating your own baremetal cluster you will need to add the service which will manage provisioning IPs for LoadBalancer type Services. One such service is Metal-LB。
安装 MetalLB 前检查 requirements.
在部署 MetalLB 之前，我们需要执行一个步骤：
If you’re using kube-proxy in IPVS mode, since Kubernetes v1.14.2 you have to enable strict ARP mode.
请注意，如果您使用 kube-router 作为服务代理，则不需要它，因为它默认启用严格的 ARP。 输入此命令：
$ kubectl edit configmap -n kube-system kube-proxy
在打开的页面中搜索 mode，在我的例子中，mode 等于空字符串，所以我不需要更改任何东西，但在这种情况下如安装指南中所述设置为 ipvs 您需要在此文件中设置以下配置：
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
strictARP: true
作为下一步，您需要 运行 这些命令：
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
在 运行 上面的命令之后，我们有这个：
$ kubectl get all -n metallb-system
NAME READY STATUS RESTARTS AGE
pod/controller-6b78sff7d9-2rv2f 1/1 Running 0 3m
pod/speaker-7bqev 1/1 Running 0 3m
pod/speaker-txrg5 1/1 Running 0 3m
pod/speaker-w7th5 1/1 Running 0 3m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 3m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/controller 1/1 1 1 3m
NAME DESIRED CURRENT READY AGE
replicaset.apps/controller-6b78sff7d9 1 1 1 3m
MetalLB 需要一些 IPv4 addresses :
$ ip a s
1: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
[...]
inet 10.240.1.59/24 brd 10.240.1.255 scope global dynamic noprefixroute ens160
valid_lft 425669sec preferred_lft 421269sec
[...]
ens160 是我的控制平面网络接口，如您所见，它的 IP 范围是 10.240.1.59/24 所以我要在这个网络中分配一组 IP 地址：
$ sipcalc 10.240.1.59/24
-[ipv4 : 10.240.1.59/24] - 0
[CIDR]
Host address - 10.240.1.59
Host address (decimal) - 183500115
Host address (hex) - AF0031B
Network address - 10.240.1.0
Network mask - 255.255.255.0
Network mask (bits) - 24
Network mask (hex) - FFFFF000
Broadcast address - 10.240.1.255
Cisco wildcard - 0.0.0.255
Addresses in network - 256
Network range - 10.240.1.0 - 10.240.1.255
Usable range - 10.240.1.1 - 10.240.1.254
现在我要从 Usable range 中获取 10 个 IP 地址并将其分配给 MetalLB。
让我们为 MetalLB 创建一个 configmap :
$ sudo nano metallb-cm.yaml
将以下配置粘贴到 metallb-cm.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.240.1.100-10.240.1.110
然后保存文件和运行这个命令：
$ kubectl create -f metallb-cm.yaml
现在让我们再次检查我们的服务：
$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default bar-service ClusterIP 10.103.49.102 <none> 5000/TCP 15m
default foo-service ClusterIP 10.102.52.89 <none> 5000/TCP 19m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
kong kong-proxy LoadBalancer 10.104.79.161 10.240.1.100 80:31583/TCP,443:30053/TCP 82m
kong kong-validation-webhook ClusterIP 10.109.75.104 <none> 443/TCP 82m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 23h
如您所见，LoadBalancer 类型的服务现在有一个 IP 地址。