TensorFlow 服务无法通过预测
TensorFlow serving unable to pass predictions
按照 https://www.tensorflow.org/tutorials/images/hub_with_keras 上的教程生成了一个 SavedModel，我想将其用于 TF 服务。
目前我无法使用 TF Serving 1.14 通过预测，我得到：
{ "error": "JSON Parse error: Invalid value. at offset: 0" }
原图：
curl -o image.jpg https://storage.googleapis.com/cloud-samples-data/ml-engine/flowers/tulips/4520577328_a94c11e806_n.jpg
Python代码：
import numpy as np
import json
from PIL import Image
def convert_to_json(image_file):
"""Open image, convert it to numpy and create JSON request"""
img = Image.open(image_file).resize(IMAGE_SHAPE)
img_array = np.array(img)
predict_request = {"keras_layer_input": [img_array.tolist()]}
with open('result.json', 'w') as fp:
json.dump(predict_request, fp)
return predict_request
prediction_data = convert_to_json('image.jpg')
TensorFlow saved_model_cli
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['keras_layer_input'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 224, 224, 3)
name: keras_layer_input:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 5)
name: dense/Softmax:0
Method name is: tensorflow/serving/predict
TensorFlow 服务器：
tensorflow_model_server --rest_api_port=8502 --model_base_path="${MODEL_DIR}"
2019-07-11 06:43:07.124549: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: default model_base_path: gs://dpe-sandbox/saved_models/
2019-07-11 06:43:07.124906: I tensorflow_serving/model_servers/server_core.cc:461] Adding/updating models.
2019-07-11 06:43:07.124925: I tensorflow_serving/model_servers/server_core.cc:558] (Re-)adding model: default
2019-07-11 06:43:09.199345: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: default version: 1}
2019-07-11 06:43:09.199478: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: default version: 1}
2019-07-11 06:43:09.199546: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: default version: 1}
2019-07-11 06:43:09.199630: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: gs://dpe-sandbox/saved_models/1
2019-07-11 06:43:09.199644: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: gs://dpe-sandbox/saved_models/1
2019-07-11 06:43:09.422041: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-07-11 06:43:09.473872: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-11 06:43:10.852346: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 06:43:10.852833: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
totalMemory: 14.73GiB freeMemory: 14.60GiB
2019-07-11 06:43:10.967813: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 06:43:10.968300: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:05.0
totalMemory: 14.73GiB freeMemory: 14.60GiB
2019-07-11 06:43:10.971176: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1
2019-07-11 06:43:11.998039: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-11 06:43:11.998112: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1
2019-07-11 06:43:11.998119: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y
2019-07-11 06:43:11.998125: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N
2019-07-11 06:43:11.998781: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14103 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
2019-07-11 06:43:11.999365: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14103 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5)
2019-07-11 06:43:12.092750: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:182] Restoring SavedModel bundle.
2019-07-11 06:43:13.436094: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:132] Running initialization op on SavedModel bundle.
2019-07-11 06:43:13.517255: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:285] SavedModel load for tags { serve }; Status: success. Took 4317544 microseconds.
2019-07-11 06:43:13.663286: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:101] No warmup data file found at gs://dpe-sandbox/saved_models/1/assets.extra/tf_serving_warmup_requests
2019-07-11 06:43:15.562663: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: default version: 1}
2019-07-11 06:43:15.582902: I tensorflow_serving/model_servers/server.cc:313] Running gRPC ModelServer at 0.0.0.0:8500 ...
2019-07-11 06:43:15.584199: I tensorflow_serving/model_servers/server.cc:333] Exporting HTTP/REST API at:localhost:8502 ...
[evhttp_server.cc : 237] RAW: Entering the event loop ...
不确定我是否正确编码图像？
我复制并粘贴了值，似乎是有效的 JSON
为了社区的利益，即使它出现在评论部分中，也要在此（回答）部分中提及解决方案。
通过在下面的代码行中将 keras_layer_input 替换为 instances 来解决问题。这是因为，对于 predict SignatureDef，在语法上，推理的输入应该表示为 instances 或 inputs.
predict_request = {"keras_layer_input": [img_array.tolist()]}
工作代码是：
predict_request = {"instances": [img_array.tolist()]}
按照 https://www.tensorflow.org/tutorials/images/hub_with_keras 上的教程生成了一个 SavedModel，我想将其用于 TF 服务。
目前我无法使用 TF Serving 1.14 通过预测，我得到：
{ "error": "JSON Parse error: Invalid value. at offset: 0" }
原图：
curl -o image.jpg https://storage.googleapis.com/cloud-samples-data/ml-engine/flowers/tulips/4520577328_a94c11e806_n.jpg
Python代码：
import numpy as np
import json
from PIL import Image
def convert_to_json(image_file):
"""Open image, convert it to numpy and create JSON request"""
img = Image.open(image_file).resize(IMAGE_SHAPE)
img_array = np.array(img)
predict_request = {"keras_layer_input": [img_array.tolist()]}
with open('result.json', 'w') as fp:
json.dump(predict_request, fp)
return predict_request
prediction_data = convert_to_json('image.jpg')
TensorFlow saved_model_cli
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['keras_layer_input'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 224, 224, 3)
name: keras_layer_input:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 5)
name: dense/Softmax:0
Method name is: tensorflow/serving/predict
TensorFlow 服务器：
tensorflow_model_server --rest_api_port=8502 --model_base_path="${MODEL_DIR}"
2019-07-11 06:43:07.124549: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: default model_base_path: gs://dpe-sandbox/saved_models/
2019-07-11 06:43:07.124906: I tensorflow_serving/model_servers/server_core.cc:461] Adding/updating models.
2019-07-11 06:43:07.124925: I tensorflow_serving/model_servers/server_core.cc:558] (Re-)adding model: default
2019-07-11 06:43:09.199345: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: default version: 1}
2019-07-11 06:43:09.199478: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: default version: 1}
2019-07-11 06:43:09.199546: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: default version: 1}
2019-07-11 06:43:09.199630: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: gs://dpe-sandbox/saved_models/1
2019-07-11 06:43:09.199644: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: gs://dpe-sandbox/saved_models/1
2019-07-11 06:43:09.422041: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-07-11 06:43:09.473872: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-11 06:43:10.852346: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 06:43:10.852833: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
totalMemory: 14.73GiB freeMemory: 14.60GiB
2019-07-11 06:43:10.967813: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-11 06:43:10.968300: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:05.0
totalMemory: 14.73GiB freeMemory: 14.60GiB
2019-07-11 06:43:10.971176: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1
2019-07-11 06:43:11.998039: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-11 06:43:11.998112: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1
2019-07-11 06:43:11.998119: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y
2019-07-11 06:43:11.998125: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N
2019-07-11 06:43:11.998781: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14103 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
2019-07-11 06:43:11.999365: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 14103 MB memory) -> physical GPU (device: 1, name: Tesla T4, pci bus id: 0000:00:05.0, compute capability: 7.5)
2019-07-11 06:43:12.092750: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:182] Restoring SavedModel bundle.
2019-07-11 06:43:13.436094: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:132] Running initialization op on SavedModel bundle.
2019-07-11 06:43:13.517255: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:285] SavedModel load for tags { serve }; Status: success. Took 4317544 microseconds.
2019-07-11 06:43:13.663286: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:101] No warmup data file found at gs://dpe-sandbox/saved_models/1/assets.extra/tf_serving_warmup_requests
2019-07-11 06:43:15.562663: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: default version: 1}
2019-07-11 06:43:15.582902: I tensorflow_serving/model_servers/server.cc:313] Running gRPC ModelServer at 0.0.0.0:8500 ...
2019-07-11 06:43:15.584199: I tensorflow_serving/model_servers/server.cc:333] Exporting HTTP/REST API at:localhost:8502 ...
[evhttp_server.cc : 237] RAW: Entering the event loop ...
不确定我是否正确编码图像？ 我复制并粘贴了值，似乎是有效的 JSON
为了社区的利益，即使它出现在评论部分中，也要在此（回答）部分中提及解决方案。
通过在下面的代码行中将 keras_layer_input 替换为 instances 来解决问题。这是因为，对于 predict SignatureDef，在语法上，推理的输入应该表示为 instances 或 inputs.
predict_request = {"keras_layer_input": [img_array.tolist()]}
工作代码是：
predict_request = {"instances": [img_array.tolist()]}