Google App Engine 部署调试 "Error Response: [4] Timed out waiting for the app infrastructure to become healthy"

Question

简而言之： 我们在 Google App Engine Nodejs 运行时、flex 环境中有 Web 应用程序运行。从 5 天前开始，我们所有的部署都开始失败，原因是：

ERROR: (gcloud.app.deploy) Error Response: [4] Timed out waiting for the app infrastructure to become healthy.

完整错误堆栈跟踪：

Updating service [default] (this may take several minutes)...\DEBUG: Operation [apps/PROJECT_ID/operations/45d6fec1-9261-41d2-943a-648976b971ed] not complete. Waiting to retry.
Updating service [default] (this may take several minutes)...-DEBUG: Operation [apps/PROJECT_ID/operations/45d6fec1-9261-41d2-943a-648976b971ed] complete. Result: {
    "metadata": {
        "user": "SOME_EMAIL@mail.com", 
        "target": "apps/PROJECT_ID/services/default/versions/release-0-6-3",
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1", 
        "insertTime": "2018-02-19T06:08:56.439Z", 
        "method": "google.appengine.v1.Versions.CreateVersion"
    }, 
    "done": true, 
    "name": "apps/PROJECT_ID/operations/45d6fec1-9261-41d2-943a-648976b971ed", 
    "error": {
        "message": "Timed out waiting for the app infrastructure to become healthy.", 
        "code": 4
    }
}
Updating service [default] (this may take several minutes)...failed.                                                                    
DEBUG: (gcloud.app.deploy) Error Response: [4] Timed out waiting for the app infrastructure to become healthy.
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 797, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 757, in Run
    resources = command_instance.Run(args)
  File "/usr/lib/google-cloud-sdk/lib/surface/app/deploy.py", line 65, in Run
    parallel_build=False)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 588, in RunDeploy
    flex_image_build_option=flex_image_build_option)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/app/deploy_util.py", line 394, in Deploy
    extra_config_settings)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/appengine_api_client.py", line 188, in DeployService
    message=message)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 246, in WaitForOperation
    sleep_ms=retry_interval)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 266, in WaitFor
    sleep_ms=sleep_ms)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py", line 222, in RetryOnResult
    if not should_retry(result, state):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/waiter.py", line 260, in _IsNotDone
    return not poller.IsDone(operation)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/api_lib/app/operations_util.py", line 171, in IsDone
    encoding.MessageToPyValue(operation.error)))
OperationError: Error Response: [4] Timed out waiting for the app infrastructure to become healthy.
ERROR: (gcloud.app.deploy) Error Response: [4] Timed out waiting for the app infrastructure to become healthy.

在那之前（3 周前）部署开始非常缓慢（5-20 分钟）。

命令提供的操作信息：

gcloud beta app operations describe OPERATION_ID

给出这个：

done: true
error:
  code: 4
  message: Timed out waiting for the app infrastructure to become healthy.
metadata:
  '@type': type.googleapis.com/google.appengine.v1.OperationMetadataV1
  endTime: '2018-02-19T06:36:02.752Z'
  insertTime: '2018-02-19T06:08:56.439Z'
  method: google.appengine.v1.Versions.CreateVersion
  target: apps/PROJECT_ID/services/default/versions/release-0-6-3
  user: SOME_EMAIL@mail.com
name: apps/PROJECT_ID/operations/45d6fec1-9261-41d2-943a-648976b971ed

关于如何获取有关操作的更多信息以及执行的操作的任何想法？

最佳，

亚历克斯

Answer 1

在撰写本文时，我还没有找到在 Ubuntu 从 CLI Google App Engine 部署期间获取更详细错误消息的方法，而不是传递详细信息选项 gcloud app deploy --verbosity=debug或使用 gcloud beta app operations describe OPERATION_ID.

获取有关操作的更多信息

但是他们找到了问题的原因： The reason for the Compute Engine VMs never becoming healthy is because your 'In-use IP addresses' quota [1] in the region of your App Engine Flexible application has reached its limit. 更多信息 - https://issuetracker.google.com/issues/73583699。并承诺在 CLI 的下一次更新中提供更好的错误消息

Google App Engine 部署调试 "Error Response: [4] Timed out waiting for the app infrastructure to become healthy"

Google App Engine deployment debug "Error Response: [4] Timed out waiting for the app infrastructure to become healthy"

deployment

debugging

google-appengine-node