重新训练开始 google 云停留在全局步骤 0

Question

我正在按照 flowers 教程在 google cloud ml 上重新训练 inception。我可以运行教程，训练，预测，就好了。

然后我用花数据集替换了我自己的测试数据集。图像数字的光学字符识别。

我的完整代码是here

labels

的字典文件

评估set

培训Set

运行来自 google 提供的最近 docker 版本。

`docker run -it -p "127.0.0.1:8080:8080" --entrypoint=/bin/bash  gcr.io/cloud-datalab/datalab:local-20161227

我可以预处理文件，并使用

提交训练作业

  # Submit training job.
gcloud beta ml jobs submit training "$JOB_ID" \
  --module-name trainer.task \
  --package-path trainer \
  --staging-bucket "$BUCKET" \
  --region us-central1 \
  -- \
  --output_path "${GCS_PATH}/training" \
  --eval_data_paths "${GCS_PATH}/preproc/eval*" \
  --train_data_paths "${GCS_PATH}/preproc/train*"

但它永远不会超过全局步骤 0。花教程在免费套餐上训练了大约 1 小时。我已经让我的训练持续了 11 个小时。没有动静。

查看stackdriver，没有任何进展。

我还尝试了一个包含 20 个训练图像和 10 个评估图像的小型玩具数据集。同样的问题。

GCS Bucket 最终看起来像这样

不出所料，我无法在 tensorboard 中可视化此日志，没有任何显示。

完整训练日志：

INFO    2017-01-10 17:22:00 +0000       unknown_task            Validating job requirements...
INFO    2017-01-10 17:22:01 +0000       unknown_task            Job creation request has been successfully validated.
INFO    2017-01-10 17:22:01 +0000       unknown_task            Job MeerkatReader_MeerkatReader_20170110_170701 is queued.
INFO    2017-01-10 17:22:07 +0000       unknown_task            Waiting for job to be provisioned.
INFO    2017-01-10 17:22:07 +0000       unknown_task            Waiting for TensorFlow to start.
INFO    2017-01-10 17:22:10 +0000       master-replica-0                Running task with arguments: --cluster={"master": ["master-d4f6-0:2222"]} --task={"type": "master", "index": 0} --job={
INFO    2017-01-10 17:22:10 +0000       master-replica-0                  "package_uris": ["gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz"],
INFO    2017-01-10 17:22:10 +0000       master-replica-0                  "python_module": "trainer.task",
INFO    2017-01-10 17:22:10 +0000       master-replica-0                  "args": ["--output_path", "gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/training", "--eval_data_paths", "gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/eval*", "--train_data_paths", "gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/train*"],
INFO    2017-01-10 17:22:10 +0000       master-replica-0                  "region": "us-central1"
INFO    2017-01-10 17:22:10 +0000       master-replica-0                } --beta
INFO    2017-01-10 17:22:10 +0000       master-replica-0                Downloading the package: gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz
INFO    2017-01-10 17:22:10 +0000       master-replica-0                Running command: gsutil -q cp gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz trainer-0.1.tar.gz
INFO    2017-01-10 17:22:12 +0000       master-replica-0                Building wheels for collected packages: trainer
INFO    2017-01-10 17:22:12 +0000       master-replica-0                creating '/tmp/tmpSgdSzOpip-wheel-/trainer-0.1-cp27-none-any.whl' and adding '.' to it
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/model.py'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/util.py'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/preprocess.py'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/task.py'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/metadata.json'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/WHEEL'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/METADATA'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                  Running setup.py bdist_wheel for trainer: finished with status 'done'
INFO    2017-01-10 17:22:12 +0000       master-replica-0                  Stored in directory: /root/.cache/pip/wheels/e8/0c/c7/b77d64796dbbac82503870c4881d606fa27e63942e07c75f0e
INFO    2017-01-10 17:22:12 +0000       master-replica-0                Successfully built trainer
INFO    2017-01-10 17:22:13 +0000       master-replica-0                Running command: python -m trainer.task --output_path gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/training --eval_data_paths gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/eval* --train_data_paths gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/train*
INFO    2017-01-10 17:22:14 +0000       master-replica-0                Starting master/0
INFO    2017-01-10 17:22:14 +0000       master-replica-0                Initialize GrpcChannelCache for job master -> {0 -> localhost:2222}
INFO    2017-01-10 17:22:14 +0000       master-replica-0                Started server with target: grpc://localhost:2222
ERROR   2017-01-10 17:22:16 +0000       master-replica-0                device_filters: "/job:ps"
INFO    2017-01-10 17:22:19 +0000       master-replica-0                global_step/sec: 0

只是重复最后一行，直到我杀掉它。

我对这项服务的心智模型不正确吗？欢迎所有建议。

Answer 1

一切看起来都很好。我怀疑你的数据有问题。具体来说，我怀疑 TF 无法从您的 GCS 文件中读取任何数据（它们是空的吗？）？因此，当您调用火车时，TF 最终会阻止尝试读取它无法读取的一批数据。

我建议在 Trainer.run_training 中围绕对 session.run 的调用添加日志语句。这将告诉您那条线是否卡住了。

我还建议检查您的 GCS 文件的大小。

TensorFlow 也有一个实验性的 RunOptions，允许您为 Session.run 指定超时。一旦此功能准备就绪，这可能有助于确保代码不会永远阻塞。

重新训练开始 google 云停留在全局步骤 0

Re-training inception google cloud stuck at global step 0

python

tensorflow

google-cloud-ml