尽管更改了 progressDeadlineSeconds，但仍将 运行 保持为 "exceeded its progress dead line"
Keep running into "exceeded its progress dead line" despite changing progressDeadlineSeconds
我是新的 AKS、ACR 和 DevOps 管道，我正在尝试设置 CI/CD 管道。
我有一个同时包含 AKS 和 ACR 的资源组设置。 AKS 正在使用 Standard_B2s，而且此时只有一个节点，因为我只是在玩。
图像在提交到 master 时自动部署到 ACR——还没有弄清楚如何设置测试——但是当涉及到部署到 AKS 时，我只是不断得到：
##[error]error: deployment "client-deployment" exceeded its progress deadline
我已经更改了我的 client.yaml 以包含大约一个小时的 progressDeadlineSeconds，因为 10、15 和 20 分钟没有用：
apiVersion: apps/v1
kind: Deployment
metadata:
name: client-deployment
spec:
progressDeadlineSeconds: 3600
replicas: 1
selector:
matchLabels:
component: client
template:
metadata:
labels:
component: client
spec:
containers:
- name: client
image: testappcontainers.azurecr.io/testapp-client
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: client-cluster-ip-service
spec:
type: ClusterIP
selector:
component: client
ports:
- port: 3000
targetPort: 3000
我刚刚修改了 Pipelines 为我生成的 azure-pipelines.yml，我目前拥有的如下：
# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key>
imageRepository: 'testapp'
containerRegistry: 'testappcontainers.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)'
tag: '$(Build.BuildId)'
imagePullSecret: <imagePullSecret_key>
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker@2
displayName: Build and push client image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)-client
dockerfile: $(dockerfilePath)/client/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
- upload: manifests
artifact: manifests
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
jobs:
- deployment: Deploy
displayName: Deploy job
pool:
vmImage: $(vmImageName)
environment: 'testapp.default'
strategy:
runOnce:
deploy:
steps:
- task: KubernetesManifest@0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
- task: KubernetesManifest@0
displayName: Deploy to Kubernetes cluster
inputs:
action: deploy
manifests: |
$(Pipeline.Workspace)/manifests/client.yaml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository):$(tag)
这也是失败任务的日志：
##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task : Deploy to Kubernetes
Description : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version : 0.162.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_<token>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers<key>-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=<token>
##[debug]<token> data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=<token>
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]<token> data authorizationType = AzureSubscription
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/
##[debug]<token> auth param serviceAccountCertificate = ***
##[debug]<token> auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]<token> data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] apply
##[debug] -f
##[debug] /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] rollout
##[debug] status
##[debug] Deployment/client-deployment
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] get
##[debug] service/client-cluster-ip-service
##[debug] -o
##[debug] json
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"
},
"creationTimestamp": "name": "client-cluster-ip-service",
"namespace": "default",
"resourceVersion": "1234045",
"selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
"uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
},
"spec": {
"clusterIP": "10.0.181.220",
"ports": [
{
"port": 3000,
"protocol": "TCP",
"targetPort": 3000
}
],
"selector": {
"component": "client"
},
"sessionAffinity": "None",
"type": "ClusterIP"
},
"status": {
"loadBalancer": {}
}
}
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster
然后在 Azure CLI 中，它显示部署已存在，但没有可用 pods:
eox-dev@Azure:~$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
client-deployment 0/1 1 0 3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name: client-deployment
Namespace: default
CreationTimestamp: Tue, 31 Dec 2019 15:50:30 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...
Selector: component=client
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: component=client
Containers:
client:
Image: testappcontainers.azurecr.io/testapp-client
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: client-deployment-5688bdc69c (1/1 replicas created)
Events: <none>
那么我做错了什么？
Error from server (BadRequest): container "client" in pod
"client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and
failing to pull image
根据我的经验，这与 imagePullSecrets 和 Kubernetes namespace 更相关。
在您的 Create imagePullSecret 和 Deploy to Kubernetes cluster 任务中，我看到您没有为任务参数提供值：namespace。这将导致创建一个名为 default 的新命名空间，因为您 未指定 命名空间。
并且，由 createSecret 操作生成的 kubernetes secret 为每个命名空间分开。总之，不同的命名空间有不同的秘密值：
Secrets are stored within a given namespace and can only be accessed
by pods within the same namespace.
现在，让我们回到您的构建编译过程。
在您的 yml 定义中，Create imagePullSecret 将为新命名空间 default 创建一个 secret，该命名空间由任务自动创建，因为您没有提供给定的命名空间值。
然后，在下一个任务Deploy to Kubernetes cluster中，由于同样的原因，这里任务将重新创建另一个新的命名空间default(注意： 这与上一个不同）。此外，您可以从日志中看到此进度：
此时，从上一个任务生成的 secret 将不适用于当前命名空间。但是，如您所知，ACR 是一个私有容器注册表，我们的系统必须验证 kubernetes secret 是否可用。
此外，在部署到 Kubernetes 集群任务中，您将存储库指定为 $(imageRepository)，这与您将映像推送到的存储库不同 $(imageRepository)-client。
这也可以在您的日志中查看：
这就是为什么你的kubernetes中没有可用的节点，你也无法拉取镜像。
为避免此问题，请确保在 KubernetesManifest@0 任务中提供 namespace 值。
- task: KubernetesManifest@0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
namespace: $(k8sNamespace)
dockerRegistryEndpoint: $(DRServiceConnection)
- task: KubernetesManifest@0
displayName: Deploy to Kubernetes cluster
inputs:
action: deploy
namespace: $(k8sNamespace)
manifests: |
$(System.ArtifactsDirectory)/manifests/deployment.yml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository)-client:$(tag)
每个命名空间的 imagePullSecrets 的秘密
即使命名空间仍然出现错误，我也遇到了类似的问题
“超过了它的进度期限
等待部署 "XX-service" 部署完成：1 个更新副本中有 0 个可用..."
由于拉取镜像失败
正在添加
"imagePullPolicy: Always" 在 .yaml 中修复了我的 rollout pull 问题
我认为可能会导致此问题（尤其是在 AKS 中）的原因是您的群集在部署之前对您的容器运行运行状况检查或活动检查。
您的集群识别出您的应用程序不是 运行 并由于 CrashLoopBackOff
而进入等待状态
这可以通过在 API 应用程序的“/”上创建默认 GET 路由来解决，returns 200。这将表明您的容器是健康的或活跃的
由于异常和就绪探测失败，我遇到了类似的问题。解决了这些问题后，rollout 就成功完成了。
就我而言，这是工作节点的问题。所以我只是尝试从我的 pod 分配到的节点中提取图像。我开始知道该节点存在一些与性能相关的问题，因此我只添加了一个高规格的新节点并将 pod 分配给该节点。部署也成功了。
我是新的 AKS、ACR 和 DevOps 管道，我正在尝试设置 CI/CD 管道。
我有一个同时包含 AKS 和 ACR 的资源组设置。 AKS 正在使用 Standard_B2s，而且此时只有一个节点，因为我只是在玩。
图像在提交到 master 时自动部署到 ACR——还没有弄清楚如何设置测试——但是当涉及到部署到 AKS 时，我只是不断得到：
##[error]error: deployment "client-deployment" exceeded its progress deadline
我已经更改了我的 client.yaml 以包含大约一个小时的 progressDeadlineSeconds，因为 10、15 和 20 分钟没有用：
apiVersion: apps/v1
kind: Deployment
metadata:
name: client-deployment
spec:
progressDeadlineSeconds: 3600
replicas: 1
selector:
matchLabels:
component: client
template:
metadata:
labels:
component: client
spec:
containers:
- name: client
image: testappcontainers.azurecr.io/testapp-client
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: client-cluster-ip-service
spec:
type: ClusterIP
selector:
component: client
ports:
- port: 3000
targetPort: 3000
我刚刚修改了 Pipelines 为我生成的 azure-pipelines.yml，我目前拥有的如下：
# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- master
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key>
imageRepository: 'testapp'
containerRegistry: 'testappcontainers.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)'
tag: '$(Build.BuildId)'
imagePullSecret: <imagePullSecret_key>
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker@2
displayName: Build and push client image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)-client
dockerfile: $(dockerfilePath)/client/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
- upload: manifests
artifact: manifests
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
jobs:
- deployment: Deploy
displayName: Deploy job
pool:
vmImage: $(vmImageName)
environment: 'testapp.default'
strategy:
runOnce:
deploy:
steps:
- task: KubernetesManifest@0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
- task: KubernetesManifest@0
displayName: Deploy to Kubernetes cluster
inputs:
action: deploy
manifests: |
$(Pipeline.Workspace)/manifests/client.yaml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository):$(tag)
这也是失败任务的日志：
##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task : Deploy to Kubernetes
Description : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version : 0.162.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_<token>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers<key>-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=<token>
##[debug]<token> data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=<token>
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]<token> data authorizationType = AzureSubscription
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/
##[debug]<token> auth param serviceAccountCertificate = ***
##[debug]<token> auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]<token> data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] apply
##[debug] -f
##[debug] /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] rollout
##[debug] status
##[debug] Deployment/client-deployment
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug] get
##[debug] service/client-cluster-ip-service
##[debug] -o
##[debug] json
##[debug] --namespace
##[debug] default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"
},
"creationTimestamp": "name": "client-cluster-ip-service",
"namespace": "default",
"resourceVersion": "1234045",
"selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
"uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
},
"spec": {
"clusterIP": "10.0.181.220",
"ports": [
{
"port": 3000,
"protocol": "TCP",
"targetPort": 3000
}
],
"selector": {
"component": "client"
},
"sessionAffinity": "None",
"type": "ClusterIP"
},
"status": {
"loadBalancer": {}
}
}
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster
然后在 Azure CLI 中，它显示部署已存在，但没有可用 pods:
eox-dev@Azure:~$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
client-deployment 0/1 1 0 3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name: client-deployment
Namespace: default
CreationTimestamp: Tue, 31 Dec 2019 15:50:30 +0000
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...
Selector: component=client
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: component=client
Containers:
client:
Image: testappcontainers.azurecr.io/testapp-client
Port: 3000/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: client-deployment-5688bdc69c (1/1 replicas created)
Events: <none>
那么我做错了什么？
Error from server (BadRequest): container "client" in pod "client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and failing to pull image
根据我的经验，这与 imagePullSecrets 和 Kubernetes namespace 更相关。
在您的 Create imagePullSecret 和 Deploy to Kubernetes cluster 任务中，我看到您没有为任务参数提供值：namespace。这将导致创建一个名为 default 的新命名空间，因为您 未指定 命名空间。
并且，由 createSecret 操作生成的 kubernetes secret 为每个命名空间分开。总之，不同的命名空间有不同的秘密值：
Secrets are stored within a given namespace and can only be accessed by pods within the same namespace.
现在，让我们回到您的构建编译过程。
在您的 yml 定义中，Create imagePullSecret 将为新命名空间 default 创建一个 secret，该命名空间由任务自动创建，因为您没有提供给定的命名空间值。
然后，在下一个任务Deploy to Kubernetes cluster中，由于同样的原因，这里任务将重新创建另一个新的命名空间default(注意： 这与上一个不同）。此外，您可以从日志中看到此进度：
此时，从上一个任务生成的 secret 将不适用于当前命名空间。但是，如您所知，ACR 是一个私有容器注册表，我们的系统必须验证 kubernetes secret 是否可用。
此外，在部署到 Kubernetes 集群任务中，您将存储库指定为 $(imageRepository)，这与您将映像推送到的存储库不同 $(imageRepository)-client。
这也可以在您的日志中查看：
这就是为什么你的kubernetes中没有可用的节点，你也无法拉取镜像。
为避免此问题，请确保在 KubernetesManifest@0 任务中提供 namespace 值。
- task: KubernetesManifest@0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
namespace: $(k8sNamespace)
dockerRegistryEndpoint: $(DRServiceConnection)
- task: KubernetesManifest@0
displayName: Deploy to Kubernetes cluster
inputs:
action: deploy
namespace: $(k8sNamespace)
manifests: |
$(System.ArtifactsDirectory)/manifests/deployment.yml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository)-client:$(tag)
每个命名空间的 imagePullSecrets 的秘密
即使命名空间仍然出现错误，我也遇到了类似的问题 “超过了它的进度期限 等待部署 "XX-service" 部署完成：1 个更新副本中有 0 个可用..."
由于拉取镜像失败
正在添加 "imagePullPolicy: Always" 在 .yaml 中修复了我的 rollout pull 问题
我认为可能会导致此问题（尤其是在 AKS 中）的原因是您的群集在部署之前对您的容器运行运行状况检查或活动检查。 您的集群识别出您的应用程序不是 运行 并由于 CrashLoopBackOff
而进入等待状态这可以通过在 API 应用程序的“/”上创建默认 GET 路由来解决，returns 200。这将表明您的容器是健康的或活跃的
由于异常和就绪探测失败，我遇到了类似的问题。解决了这些问题后，rollout 就成功完成了。
就我而言，这是工作节点的问题。所以我只是尝试从我的 pod 分配到的节点中提取图像。我开始知道该节点存在一些与性能相关的问题，因此我只添加了一个高规格的新节点并将 pod 分配给该节点。部署也成功了。