是否可以在 Kubernetes 中启动自签名 Docker 注册表并让其他服务使用它作为注册表来获取其映像?
Is it possible to start Self-Signed Docker Registry in Kubernetes and have other service use that as the registry to get its image?
问题陈述
- 我想提供一个私有注册表,其中捆绑了我的产品所需的所有图像(是的,它将是胖图像,但我可以接受)
- 我会以某种方式手动上传这张图片
- 我会 运行 docker 私有注册表作为 Kubernetes 中的一项服务(可能在某些名称空间中)
- 当其他services/deployments(与registry在同一个命名空间)发生在Kubernetes中时,他们应该使用一致的名称引用这个registry
约束
- 我们希望注册表只暴露给集群而不是外部
- 我们想使用自签名证书而不是由 CA 签名
我遵循了这些链接中的一些说明(不知道这样做是否正确)
- https://kubernetes.io/docs/concepts/cluster-administration/certificates/
- https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request-object-to-send-to-the-kubernetes-api
创建通过 Kubernetes 签名的证书
创建 server.key
创建一个csr.info
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = US
ST = oh
L = cincinnati
O = engg
OU = prod
CN = prateek.svc.cluster.local
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = registry.prateek.svc.cluster.local
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
创建了 server.csr (openssl req -new -key server.key -out server.csr -config csr.conf)
在K8s中创建CertificateSigningRequest
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: registry.prateek
spec:
groups:
- system:authenticated
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
- 检查 CSR 是否存在
kubectl describe csr registry.prateek
Name: registry.prateek
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certificates.k8s.io/v1beta1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"registry.prateek","namespace":""},"spec":{"groups":["system:authenticated"],"request":"LS0sdfsfsdsfd=","usages":["digital signature","key encipherment","server auth"]}}
CreationTimestamp: Thu, 11 Apr 2019 11:15:42 -0400
Requesting User: docker-for-desktop
Status: Pending
Subject:
Common Name: prateek.svc.cluster.local
Serial Number:
Organization: engg
Organizational Unit: prod
Country: US
Locality: cincinnati
Province: oh
Subject Alternative Names:
DNS Names: registry.prateek.svc.cluster.local
Events: <none>
- 批准了 CSR:kubectl certificate approve registry.prateek
启动注册表内部服务
- 为种类添加了证书和密钥:Secret
注册表-secret.yml
apiVersion: v1
kind: Secret
metadata:
name: registry-credentials
data:
certificate: <CERTIFICATE in base64>
key: <KEY in base64>
- 创建注册表部署和服务(使用这些秘密)
注册表-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: prateek
labels:
app: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: prateek/registry
imagePullPolicy: IfNotPresent
ports:
- containerPort: 443
env:
- name: REGISTRY_HTTP_ADDR
value: "0.0.0.0:443"
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: "/certs/certificate"
- name: REGISTRY_HTTP_TLS_KEY
value: "/certs/key"
volumeMounts:
- name: cert-files
mountPath: /certs
volumes:
- name: cert-files
secret:
secretName: registry-credentials
注册表-service.yml
apiVersion: v1
kind: Service
metadata:
name: registry
namespace: prateek
spec:
selector:
app: registry
ports:
- protocol: TCP
port: 443
targetPort: 443
type: LoadBalancer
测试注册服务已启动
- 尝试通过测试容器访问注册表端点。我已经在 docker 中加载了这个测试舱的图像。
curl https://registry.prateek.svc.cluster.local/v2/_catalog -k
{"repositories":["prateek/echo"]}
使用注册表服务中的映像进行部署
- 尝试使用镜像部署:registry。prateek/prateek/echo:latest
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: cequence
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: registry.prateek/prateek/echo:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5678
args: ["-text=hello"]
- 部署给出错误
Normal Pulling 10s (x2 over 25s) kubelet, docker-for-desktop pulling image "registry.prateek/prateek/echo:latest"
Warning Failed 10s (x2 over 25s) kubelet, docker-for-desktop Failed to pull image "registry.prateek/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek/v2/: Service Unavailable
- 将部署更改为具有图像:registry.prateek.svc.cluster.local/prateek/echo:latest
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: cequence
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: registry.prateek.svc.cluster.local/prateek/echo:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5678
args: ["-text=hello"]
- 得到类似的错误
Warning Failed 1s kubelet, docker-for-desktop Failed to pull image "registry.prateek.svc.cluster.local/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek.svc.cluster.local/v2/: Service Unavailable
我什至不知道这是可能的。 运行 docker 注册表作为服务,并指向命名空间中的其他服务以在集群中使用该注册表部署。欢迎任何建议
容器守护进程 运行 在 kubernetes 之外。
因此,如果你想拉取镜像,你需要确保registry可以从节点直接访问,而不是像服务一样使用kubernetes机制。 (不像你在步骤9中通过pod测试的,你必须能够直接在节点上工作!)
通常的选择是创建一个 DNS 条目或 hosts.txt 条目以指向一个节点,其中通过 hostPort
(容器)或 nodePort
(服务)注册表是可访问或您使用适当的入口。
问题陈述
- 我想提供一个私有注册表,其中捆绑了我的产品所需的所有图像(是的,它将是胖图像,但我可以接受)
- 我会以某种方式手动上传这张图片
- 我会 运行 docker 私有注册表作为 Kubernetes 中的一项服务(可能在某些名称空间中)
- 当其他services/deployments(与registry在同一个命名空间)发生在Kubernetes中时,他们应该使用一致的名称引用这个registry
约束
- 我们希望注册表只暴露给集群而不是外部
- 我们想使用自签名证书而不是由 CA 签名
我遵循了这些链接中的一些说明(不知道这样做是否正确)
- https://kubernetes.io/docs/concepts/cluster-administration/certificates/
- https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request-object-to-send-to-the-kubernetes-api
创建通过 Kubernetes 签名的证书
创建 server.key
创建一个csr.info
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
C = US
ST = oh
L = cincinnati
O = engg
OU = prod
CN = prateek.svc.cluster.local
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = registry.prateek.svc.cluster.local
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
创建了 server.csr (openssl req -new -key server.key -out server.csr -config csr.conf)
在K8s中创建CertificateSigningRequest
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: registry.prateek
spec:
groups:
- system:authenticated
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
- 检查 CSR 是否存在
kubectl describe csr registry.prateek
Name: registry.prateek
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certificates.k8s.io/v1beta1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"registry.prateek","namespace":""},"spec":{"groups":["system:authenticated"],"request":"LS0sdfsfsdsfd=","usages":["digital signature","key encipherment","server auth"]}}
CreationTimestamp: Thu, 11 Apr 2019 11:15:42 -0400
Requesting User: docker-for-desktop
Status: Pending
Subject:
Common Name: prateek.svc.cluster.local
Serial Number:
Organization: engg
Organizational Unit: prod
Country: US
Locality: cincinnati
Province: oh
Subject Alternative Names:
DNS Names: registry.prateek.svc.cluster.local
Events: <none>
- 批准了 CSR:kubectl certificate approve registry.prateek
启动注册表内部服务
- 为种类添加了证书和密钥:Secret
apiVersion: v1
kind: Secret
metadata:
name: registry-credentials
data:
certificate: <CERTIFICATE in base64>
key: <KEY in base64>
- 创建注册表部署和服务(使用这些秘密) 注册表-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
namespace: prateek
labels:
app: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: prateek/registry
imagePullPolicy: IfNotPresent
ports:
- containerPort: 443
env:
- name: REGISTRY_HTTP_ADDR
value: "0.0.0.0:443"
- name: REGISTRY_HTTP_TLS_CERTIFICATE
value: "/certs/certificate"
- name: REGISTRY_HTTP_TLS_KEY
value: "/certs/key"
volumeMounts:
- name: cert-files
mountPath: /certs
volumes:
- name: cert-files
secret:
secretName: registry-credentials
注册表-service.yml
apiVersion: v1
kind: Service
metadata:
name: registry
namespace: prateek
spec:
selector:
app: registry
ports:
- protocol: TCP
port: 443
targetPort: 443
type: LoadBalancer
测试注册服务已启动
- 尝试通过测试容器访问注册表端点。我已经在 docker 中加载了这个测试舱的图像。
curl https://registry.prateek.svc.cluster.local/v2/_catalog -k
{"repositories":["prateek/echo"]}
使用注册表服务中的映像进行部署
- 尝试使用镜像部署:registry。prateek/prateek/echo:latest
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: cequence
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: registry.prateek/prateek/echo:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5678
args: ["-text=hello"]
- 部署给出错误
Normal Pulling 10s (x2 over 25s) kubelet, docker-for-desktop pulling image "registry.prateek/prateek/echo:latest"
Warning Failed 10s (x2 over 25s) kubelet, docker-for-desktop Failed to pull image "registry.prateek/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek/v2/: Service Unavailable
- 将部署更改为具有图像:registry.prateek.svc.cluster.local/prateek/echo:latest
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: cequence
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: registry.prateek.svc.cluster.local/prateek/echo:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5678
args: ["-text=hello"]
- 得到类似的错误
Warning Failed 1s kubelet, docker-for-desktop Failed to pull image "registry.prateek.svc.cluster.local/prateek/echo:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.prateek.svc.cluster.local/v2/: Service Unavailable
我什至不知道这是可能的。 运行 docker 注册表作为服务,并指向命名空间中的其他服务以在集群中使用该注册表部署。欢迎任何建议
容器守护进程 运行 在 kubernetes 之外。
因此,如果你想拉取镜像,你需要确保registry可以从节点直接访问,而不是像服务一样使用kubernetes机制。 (不像你在步骤9中通过pod测试的,你必须能够直接在节点上工作!)
通常的选择是创建一个 DNS 条目或 hosts.txt 条目以指向一个节点,其中通过 hostPort
(容器)或 nodePort
(服务)注册表是可访问或您使用适当的入口。