cloudml 再培训开始 - 收到超出有效范围的标签值
cloudml retraining inception - received a label value outside the valid range
我正在跟花tutorial在google cloud ml上重新训练inception。我可以 运行 教程，训练，预测，就好了。
然后我用花数据集替换了我自己的测试数据集。图像数字的光学字符识别。
训练模型时出现错误：
Invalid argument: Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4
训练和评估数据集的格式如下所示：
root@e925cd9502c0:~/MeerkatReader/cloudML# head training_dataGCS.csv
gs://api-project-773889352370-ml/TrainingData/0_2.jpg,H
gs://api-project-773889352370-ml/TrainingData/0_4.jpg,One
gs://api-project-773889352370-ml/TrainingData/0_5.jpg,Five
字典文件看起来像这样
$ cat cloudML/dict.txt
Eight
F
Five
Forward_slash
Four
H
Nine
One
Seven
Six
Three
Two
Zero
我最初有 1,2,3,4 和 / 等标签，但我将它们更改为字符串，以防它们是特殊字符（尤其是 /）。我可以看到一条有点类似的消息 ，但这与基于 0 的索引有关。
消息奇怪的是，确实有13种标签类型。不知何故，tensorflow 只寻找 7 (0-6)。我的问题是，什么样的格式错误可能会使 tensorflow 认为标签比实际标签少。我可以确认 80-20 分割的测试数据和训练数据都具有所有标签 类（尽管频率不同）。
我 运行正在从 google 提供的最近 docker 构建中获取信息。
`docker run -it -p "127.0.0.1:8080:8080" --entrypoint=/bin/bash gcr.io/cloud-datalab/datalab:local-20161227
我正在使用
提交训练作业
# Submit training job.
gcloud beta ml jobs submit training "$JOB_ID" \
--module-name trainer.task \
--package-path trainer \
--staging-bucket "$BUCKET" \
--region us-central1 \
-- \
--output_path "${GCS_PATH}/training" \
--eval_data_paths "${GCS_PATH}/preproc/eval*" \
--train_data_paths "${GCS_PATH}/preproc/train*"
完整错误：
Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4 [[Node: evaluate/xentropy/xentropy = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT64, _device="/job:master/replica:0/task:0/cpu:0"](final_ops/input/Wx_plus_b/fully_connected_1/BiasAdd, inputs/Squeeze)]] Caused by op u'evaluate/xentropy/xentropy', defined at: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 545, in <module> tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run sys.exit(main(sys.argv[:1] + flags_passthrough)) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 308, in main run(model, argv) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 439, in run dispatch(args, model, cluster, task) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 480, in dispatch Trainer(args, model, cluster, task).run_training() File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 187, in run_training self.args.batch_size) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 278, in build_train_graph return self.build_graph(data_paths, batch_size, GraphMod.TRAIN) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 256, in build_graph loss_value = loss(logits, labels) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 396, in loss logits, labels, name='xentropy') File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1544, in sparse_softmax_cross_entropy_with_logits precise_logits, labels, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 2376, in _sparse_softmax_cross_entropy_with_logits features=features, labels=labels, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2238, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1130, in __init__ self._traceback = _extract_stack() InvalidArgumentError (see above for traceback): Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4 [[Node: evaluate/xentropy/xentropy = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT64, _device="/job:master/replica:0/task:0/cpu:0"](final_ops/input/Wx_plus_b/fully_connected_1/BiasAdd, inputs/Squeeze)]]
我的存储桶中的一切看起来都很好
并保存我的日志事件。
我认为您需要在提交训练作业时指定 --label_count 13。该标志应该在 -- 之后的第二组标志之后，因为它需要传递给您正在执行的代码，而不是 gcloud/Cloud ML.
问题在于 TensorFlow 训练代码在开始逐步处理数据之前需要知道要生成多少输出 logit；因此它无法检查预处理步骤中的中间文件。
如果有帮助请告诉我。
-- 特定型号之后的标志在 model.py 文件中指定：
def create_model():
"""Factory method that creates model to be used by generic task.py."""
parser = argparse.ArgumentParser()
# Label count needs to correspond to nubmer of labels in dictionary used
# during preprocessing.
parser.add_argument('--label_count', type=int, default=5)
parser.add_argument('--dropout', type=float, default=0.5)
parser.add_argument(
'--inception_checkpoint_file',
type=str,
default=DEFAULT_INCEPTION_CHECKPOINT)
args, task_args = parser.parse_known_args()
override_if_not_in_args('--max_steps', '1000', task_args)
override_if_not_in_args('--batch_size', '100', task_args)
override_if_not_in_args('--eval_set_size', '370', task_args)
override_if_not_in_args('--eval_interval_secs', '2', task_args)
override_if_not_in_args('--log_interval_secs', '2', task_args)
override_if_not_in_args('--min_train_eval_rate', '2', task_args)
return Model(args.label_count, args.dropout,
args.inception_checkpoint_file), task_args
请注意，您可以更改 label_count、dropout、max_steps 等内容来影响模型训练。
HTH.
我正在跟花tutorial在google cloud ml上重新训练inception。我可以 运行 教程，训练，预测，就好了。
然后我用花数据集替换了我自己的测试数据集。图像数字的光学字符识别。
训练模型时出现错误：
Invalid argument: Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4
训练和评估数据集的格式如下所示：
root@e925cd9502c0:~/MeerkatReader/cloudML# head training_dataGCS.csv
gs://api-project-773889352370-ml/TrainingData/0_2.jpg,H
gs://api-project-773889352370-ml/TrainingData/0_4.jpg,One
gs://api-project-773889352370-ml/TrainingData/0_5.jpg,Five
字典文件看起来像这样
$ cat cloudML/dict.txt
Eight
F
Five
Forward_slash
Four
H
Nine
One
Seven
Six
Three
Two
Zero
我最初有 1,2,3,4 和 / 等标签，但我将它们更改为字符串，以防它们是特殊字符（尤其是 /）。我可以看到一条有点类似的消息
消息奇怪的是，确实有13种标签类型。不知何故，tensorflow 只寻找 7 (0-6)。我的问题是，什么样的格式错误可能会使 tensorflow 认为标签比实际标签少。我可以确认 80-20 分割的测试数据和训练数据都具有所有标签 类（尽管频率不同）。
我 运行正在从 google 提供的最近 docker 构建中获取信息。
`docker run -it -p "127.0.0.1:8080:8080" --entrypoint=/bin/bash gcr.io/cloud-datalab/datalab:local-20161227
我正在使用
提交训练作业 # Submit training job.
gcloud beta ml jobs submit training "$JOB_ID" \
--module-name trainer.task \
--package-path trainer \
--staging-bucket "$BUCKET" \
--region us-central1 \
-- \
--output_path "${GCS_PATH}/training" \
--eval_data_paths "${GCS_PATH}/preproc/eval*" \
--train_data_paths "${GCS_PATH}/preproc/train*"
完整错误：
Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4 [[Node: evaluate/xentropy/xentropy = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT64, _device="/job:master/replica:0/task:0/cpu:0"](final_ops/input/Wx_plus_b/fully_connected_1/BiasAdd, inputs/Squeeze)]] Caused by op u'evaluate/xentropy/xentropy', defined at: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 545, in <module> tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 43, in run sys.exit(main(sys.argv[:1] + flags_passthrough)) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 308, in main run(model, argv) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 439, in run dispatch(args, model, cluster, task) File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 480, in dispatch Trainer(args, model, cluster, task).run_training() File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 187, in run_training self.args.batch_size) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 278, in build_train_graph return self.build_graph(data_paths, batch_size, GraphMod.TRAIN) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 256, in build_graph loss_value = loss(logits, labels) File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line 396, in loss logits, labels, name='xentropy') File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1544, in sparse_softmax_cross_entropy_with_logits precise_logits, labels, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 2376, in _sparse_softmax_cross_entropy_with_logits features=features, labels=labels, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2238, in create_op original_op=self._default_original_op, op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1130, in __init__ self._traceback = _extract_stack() InvalidArgumentError (see above for traceback): Received a label value of 13 which is outside the valid range of [0, 6). Label values: 6 3 2 7 3 7 6 6 12 6 5 2 3 6 8 8 8 8 4 6 5 13 7 4 8 12 5 2 4 12 12 8 8 8 12 6 4 2 12 4 3 8 2 6 8 12 2 8 4 6 2 4 12 5 5 7 6 2 2 3 2 8 2 5 2 8 2 7 4 12 8 4 2 4 8 2 2 8 2 8 7 6 8 3 5 5 5 8 8 2 5 3 9 8 5 8 3 2 5 4 [[Node: evaluate/xentropy/xentropy = SparseSoftmaxCrossEntropyWithLogits[T=DT_FLOAT, Tlabels=DT_INT64, _device="/job:master/replica:0/task:0/cpu:0"](final_ops/input/Wx_plus_b/fully_connected_1/BiasAdd, inputs/Squeeze)]]
我的存储桶中的一切看起来都很好
并保存我的日志事件。
我认为您需要在提交训练作业时指定 --label_count 13。该标志应该在 -- 之后的第二组标志之后，因为它需要传递给您正在执行的代码，而不是 gcloud/Cloud ML.
问题在于 TensorFlow 训练代码在开始逐步处理数据之前需要知道要生成多少输出 logit；因此它无法检查预处理步骤中的中间文件。
如果有帮助请告诉我。
-- 特定型号之后的标志在 model.py 文件中指定：
def create_model():
"""Factory method that creates model to be used by generic task.py."""
parser = argparse.ArgumentParser()
# Label count needs to correspond to nubmer of labels in dictionary used
# during preprocessing.
parser.add_argument('--label_count', type=int, default=5)
parser.add_argument('--dropout', type=float, default=0.5)
parser.add_argument(
'--inception_checkpoint_file',
type=str,
default=DEFAULT_INCEPTION_CHECKPOINT)
args, task_args = parser.parse_known_args()
override_if_not_in_args('--max_steps', '1000', task_args)
override_if_not_in_args('--batch_size', '100', task_args)
override_if_not_in_args('--eval_set_size', '370', task_args)
override_if_not_in_args('--eval_interval_secs', '2', task_args)
override_if_not_in_args('--log_interval_secs', '2', task_args)
override_if_not_in_args('--min_train_eval_rate', '2', task_args)
return Model(args.label_count, args.dropout,
args.inception_checkpoint_file), task_args
请注意，您可以更改 label_count、dropout、max_steps 等内容来影响模型训练。
HTH.