无法访问 Kubernetes 集群上的新 Harbor 部署
Unable to Access New Harbor Deployment on Kubernetes Cluster
第一次试用 VMWare 的 Harbor 注册表服务器并将其作为部署在新的 Kubernetes 集群上进行尝试。
按照Harbor on Kubernetesguide，所有Harbor资源都申请到了k8s集群上，可以看到运行ok了。但是，我目前无法从网络浏览器访问 Harbor ui（我刚返回 "Unable to connect"）。我的猜测是安全设置不正确，某些东西丢失或在错误的地方？
make/harbor.cfg 文件配置为：
hostname = k8s-dp-2 # 这是Harbor所在的worker节点 运行..
ui_url_protocol = https
ssl_cert = /path/to/cert/on/host/harbor.crt
ssl_cert_key = /path/to/cert/on/host/harbor.key
secretkey_path = /data
我假设上述证书的路径是主机上的路径，python 脚本将从该路径获取文件，然后执行 YAML builds？
----更新---
经过评论中的建议，我现在已经在k8s集群中配置了一个nginx ingress controller。添加这个入口控制器后，我更新了 Harbor 配置以使用 http 而不再使用 https，因为 https 部分现在应该由 nginx 入口控制器处理。但是，现在有了这些配置更改，我仍然无法通过 https 访问 Harbor 服务，但我现在可以通过 kubernetes 集群的 http 端口调用它来访问 Harbor 服务。请参阅下面的测试
# kubectl get svc -n=nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress NodePort 10.103.165.23 <none> 80:31819/TCP,443:30435/TCP 20h
测试调用 1：
$ curl https://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 443: Connection refused
测试调用 2：
$ curl https://k8s-dp-2:30435/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
测试调用 3：
$ curl http://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 80: Connection refused
测试调用 4：
$ curl http://k8s-dp-2:31819/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 810 100 810 0 0 12857 0 --:--:-- --:--:-- --:--:-- 12857<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Harbor</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
</head>
<body style="overflow-y: hidden;">
...
尝试各种不同的配置后，下面发布的 YAML 配置对我有用：
入口控制器 YAML：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --default-ssl-certificate=$(POD_NAMESPACE)/default-tls-secret
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
入口 YAML：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: harbor
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
tls:
- hosts:
- k8s-dp-2
rules:
- host: k8s-dp-2
http:
paths:
- path: /
backend:
serviceName: ui
servicePort: 80
- path: /v2
backend:
serviceName: registry
servicePort: repo
- path: /service
backend:
serviceName: ui
servicePort: 80
服务 YAML：
apiVersion: v1
kind: Service
metadata:
name: ui
spec:
ports:
- port: 80
selector:
name: ui-apps
然而，找到一个可行的解决方案并不简单。必须学习很多关于入口控制器、入口等的知识。此外，我最初混合了来自两个不同工作方式不同的 nginx 入口控制器图像的配置（下面的配置适用于 quay.io 的 nginx 入口控制器）。另外，由于我仍然没有正确理解的原因，最终配置仅在完全重启所涉及的 k8s 节点后才开始工作。
第一次试用 VMWare 的 Harbor 注册表服务器并将其作为部署在新的 Kubernetes 集群上进行尝试。
按照Harbor on Kubernetesguide，所有Harbor资源都申请到了k8s集群上，可以看到运行ok了。但是，我目前无法从网络浏览器访问 Harbor ui（我刚返回 "Unable to connect"）。我的猜测是安全设置不正确，某些东西丢失或在错误的地方？
make/harbor.cfg 文件配置为：
hostname = k8s-dp-2 # 这是Harbor所在的worker节点 运行..
ui_url_protocol = https
ssl_cert = /path/to/cert/on/host/harbor.crt
ssl_cert_key = /path/to/cert/on/host/harbor.key
secretkey_path = /data
我假设上述证书的路径是主机上的路径，python 脚本将从该路径获取文件，然后执行 YAML builds？
----更新---
经过评论中的建议，我现在已经在k8s集群中配置了一个nginx ingress controller。添加这个入口控制器后，我更新了 Harbor 配置以使用 http 而不再使用 https，因为 https 部分现在应该由 nginx 入口控制器处理。但是，现在有了这些配置更改，我仍然无法通过 https 访问 Harbor 服务，但我现在可以通过 kubernetes 集群的 http 端口调用它来访问 Harbor 服务。请参阅下面的测试
# kubectl get svc -n=nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress NodePort 10.103.165.23 <none> 80:31819/TCP,443:30435/TCP 20h
测试调用 1：
$ curl https://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 443: Connection refused
测试调用 2：
$ curl https://k8s-dp-2:30435/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
测试调用 3：
$ curl http://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 80: Connection refused
测试调用 4：
$ curl http://k8s-dp-2:31819/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 810 100 810 0 0 12857 0 --:--:-- --:--:-- --:--:-- 12857<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Harbor</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
</head>
<body style="overflow-y: hidden;">
...
尝试各种不同的配置后，下面发布的 YAML 配置对我有用：
入口控制器 YAML：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --default-ssl-certificate=$(POD_NAMESPACE)/default-tls-secret
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
入口 YAML：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: harbor
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
tls:
- hosts:
- k8s-dp-2
rules:
- host: k8s-dp-2
http:
paths:
- path: /
backend:
serviceName: ui
servicePort: 80
- path: /v2
backend:
serviceName: registry
servicePort: repo
- path: /service
backend:
serviceName: ui
servicePort: 80
服务 YAML：
apiVersion: v1
kind: Service
metadata:
name: ui
spec:
ports:
- port: 80
selector:
name: ui-apps
然而，找到一个可行的解决方案并不简单。必须学习很多关于入口控制器、入口等的知识。此外，我最初混合了来自两个不同工作方式不同的 nginx 入口控制器图像的配置（下面的配置适用于 quay.io 的 nginx 入口控制器）。另外，由于我仍然没有正确理解的原因，最终配置仅在完全重启所涉及的 k8s 节点后才开始工作。