TFX Pipeline Error While Executing TFMA: AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'
TFX Pipeline Error While Executing TFMA: AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'
基本上我只重用了 iris utils and iris pipeline 中的代码，对服务输入进行了微小的更改：
def _get_serve_tf_examples_fn(model, tf_transform_output):
model.tft_layer = tf_transform_output.transform_features_layer()
feature_spec = tf_transform_output.raw_feature_spec()
print(feature_spec)
feature_spec.pop(_LABEL_KEY)
@tf.function
def serve_tf_examples_fn(*args):
parsed_features = {}
for arg in args:
parsed_features[arg.name.split(":")[0]] = arg
print(parsed_features)
transformed_features = model.tft_layer(parsed_features)
return model(transformed_features)
def run_fn(fn_args: TrainerFnArgs):
...
feature_spec = tf_transform_output.raw_feature_spec()
feature_spec.pop(_LABEL_KEY)
inputs = [tf.TensorSpec(
shape=[None, 1],
dtype=feature_spec[f].dtype,
name=f) for f in feature_spec]
signatures = {
'serving_default':
_get_serve_tf_examples_fn(model, tf_transform_output).get_concrete_function(*inputs),
}
model.save(fn_args.serving_model_dir, save_format='tf', signatures=signatures)
虹膜代码的 get_concrete_function() 原始输入只是一个带有 dtype 字符串的 TensorSpec。我已经尝试使用准确的输入为模型提供服务，但是当我测试 REST API 时，出现了解析错误。所以我尝试更改服务输入，以便它可以接收 JSON 这样的输入：
{"instances": [{"feat1": 90, "feat2": 23.8, "feat3": 12}]}
当我 运行 管道时，训练是成功的，但是当 运行 评估器组件时出现错误。这是最新的日志：
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/67/serving_model_dir as candidate model.
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/14/serving_model_dir as baseline model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:We decided to produce LargeList and LargeBinary types.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0e44560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77f8a70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
...
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in _execute
response = task()
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 313, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 483, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 518, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 983, in process_bundle
element.data)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 219, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 330, in apache_beam.runners.worker.operations.Operation.output
...
File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors' [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractBatchPredictions/Predict']
...
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0273050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77fc170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes arg
我认为评估器组件与提供输入功能没有任何关系，因为它只是将新训练的模型与最新发布的模型进行比较，但我哪里出错了？
所以最后我弄错了求值器组件，或者更恰当地说，如果我改为使用 TFMA。它确实使用了服务签名中定义的服务输入函数。根据this link，TFMA EvalConfig 使用的默认签名是“serving_default”，它描述了要序列化示例的服务模型输入。这就是为什么当我更改字符串以外的输入签名时，TFMA 会引发异常。
我认为这个签名不打算用于通过 REST API 为模型提供服务，因为仍然需要“serving_default”签名，我没有心情修改EvalConfig，我创建了另一个签名，它将接收我想要的 JSON 输入。为了让它工作，我需要创建另一个由@tf.function 修饰的函数。就这样。希望我的回答能对遇到类似问题的人有所帮助。
基本上我只重用了 iris utils and iris pipeline 中的代码，对服务输入进行了微小的更改：
def _get_serve_tf_examples_fn(model, tf_transform_output):
model.tft_layer = tf_transform_output.transform_features_layer()
feature_spec = tf_transform_output.raw_feature_spec()
print(feature_spec)
feature_spec.pop(_LABEL_KEY)
@tf.function
def serve_tf_examples_fn(*args):
parsed_features = {}
for arg in args:
parsed_features[arg.name.split(":")[0]] = arg
print(parsed_features)
transformed_features = model.tft_layer(parsed_features)
return model(transformed_features)
def run_fn(fn_args: TrainerFnArgs):
...
feature_spec = tf_transform_output.raw_feature_spec()
feature_spec.pop(_LABEL_KEY)
inputs = [tf.TensorSpec(
shape=[None, 1],
dtype=feature_spec[f].dtype,
name=f) for f in feature_spec]
signatures = {
'serving_default':
_get_serve_tf_examples_fn(model, tf_transform_output).get_concrete_function(*inputs),
}
model.save(fn_args.serving_model_dir, save_format='tf', signatures=signatures)
虹膜代码的 get_concrete_function() 原始输入只是一个带有 dtype 字符串的 TensorSpec。我已经尝试使用准确的输入为模型提供服务，但是当我测试 REST API 时，出现了解析错误。所以我尝试更改服务输入，以便它可以接收 JSON 这样的输入：
{"instances": [{"feat1": 90, "feat2": 23.8, "feat3": 12}]}
当我 运行 管道时，训练是成功的，但是当 运行 评估器组件时出现错误。这是最新的日志：
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/67/serving_model_dir as candidate model.
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/14/serving_model_dir as baseline model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:We decided to produce LargeList and LargeBinary types.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0e44560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77f8a70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
...
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in _execute
response = task()
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 313, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 483, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 518, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 983, in process_bundle
element.data)
File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 219, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 330, in apache_beam.runners.worker.operations.Operation.output
...
File "apache_beam/runners/common.py", line 1294, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
raise exc.with_traceback(traceback)
File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 570, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py", line 466, in process
result = self._batch_reducible_process(element)
File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py", line 164, in _batch_reducible_process
self._tensor_adapter.ToBatchTensors(record_batch), input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors' [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractBatchPredictions/Predict']
...
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0273050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77fc170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes arg
我认为评估器组件与提供输入功能没有任何关系，因为它只是将新训练的模型与最新发布的模型进行比较，但我哪里出错了？
所以最后我弄错了求值器组件，或者更恰当地说，如果我改为使用 TFMA。它确实使用了服务签名中定义的服务输入函数。根据this link，TFMA EvalConfig 使用的默认签名是“serving_default”，它描述了要序列化示例的服务模型输入。这就是为什么当我更改字符串以外的输入签名时，TFMA 会引发异常。
我认为这个签名不打算用于通过 REST API 为模型提供服务，因为仍然需要“serving_default”签名，我没有心情修改EvalConfig，我创建了另一个签名，它将接收我想要的 JSON 输入。为了让它工作，我需要创建另一个由@tf.function 修饰的函数。就这样。希望我的回答能对遇到类似问题的人有所帮助。