OpenVINO 如何处理 CT 分割
How can OpenVINO deal with CT segmentation
我尝试使用 COVID-19 CT segmentation dataset 和 NCS2 进行 CT 切片分割。
我引用 this GitHub 并使用其 pth 文件转换为 ONNX。
我用python mo_onnx.py --input_model model.onnx --input_shape [1,1,512,512] --data_type FP16生成IR文件。
我预期的推理结果如下图所示。
我在openvino_2021.4.689版本中试过segmentation_demo.py，但是报错如下。
[ INFO ] Initializing Inference Engine...
[ INFO ] Loading network...
[ INFO ] Reading network from IR...
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 270, in <module>
sys.exit(main() or 0)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 176, in main
model, visualizer = get_model(ie, args)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 160, in get_model
return SegmentationModel(ie, args.model), SegmentationVisualizer(args.colors)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\open_model_zoo\demos\common\python\models\segmentation.py", line 27, in __init__
self.input_blob_name = self.prepare_inputs()
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\open_model_zoo\demos\common\python\models\segmentation.py", line 43, in prepare_inputs
raise RuntimeError("3-channel 4-dimensional model's input is expected")
RuntimeError: 3-channel 4-dimensional model's input is expected
然后，我也尝试运行classification_sample_async.py，遇到下面类似的问题。
[ INFO ] Creating Inference Engine
[ INFO ] Reading the network: C:\Users\Hsien\Desktop\Segmentation-COVID-19-master\Sample_Testing\model.xml
[ INFO ] Configuring input and output blobs
[ INFO ] Loading the model to the plugin
[ INFO ] Starting inference in asynchronous mode
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\inference_engine\samples\python\classification_sample_async\classification_sample_async.py", line 168, in <module>
sys.exit(main())
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\inference_engine\samples\python\classification_sample_async\classification_sample_async.py", line 111, in main
exec_net.requests[i].async_infer({input_blob: input_data[i]})
File "ie_api.pyx", line 1222, in openvino.inference_engine.ie_api.InferRequest.async_infer
File "ie_api.pyx", line 1224, in openvino.inference_engine.ie_api.InferRequest.async_infer
File "ie_api.pyx", line 1363, in openvino.inference_engine.ie_api.InferRequest._fill_inputs
ValueError: could not broadcast input array from shape (1,3,512,512) into shape (1,1,512,512)
两条消息都可以看出，演示代码不支持灰度图
但是如何正确处理灰度CT切片分割呢？
我还提到了 210-ct-scan-live-inference.ipynb 和 110-ct-segmentation-quantize.ipynb 在 openvino_notebooks 上 GitHub 但没有成功。
（我的环境是Windows11，版本是openvino_2021.4.689。）
更新新的测试结果。
我在下面开发了一个简单的推理代码。
from openvino.inference_engine import IECore, Blob, TensorDesc
import cv2
import numpy as np
IMG_PATH = "20.jpg"
XML_PATH = "model.xml"
BIN_PATH = "model.bin"
ie_core_handler = IECore()
network = ie_core_handler.read_network(model=XML_PATH, weights=BIN_PATH)
input_blob_name = next(iter(network.input_info))
network.input_info[input_blob_name].precision = 'FP16'
executable_network = ie_core_handler.load_network(network, device_name='MYRIAD', num_requests=1)
inference_request = executable_network.requests[0]
image = cv2.imread(IMG_PATH)
re_img = cv2.resize(src=image, dsize=(512, 512))
input_data = np.expand_dims(np.transpose(re_img, (2, 0, 1)), 0).astype(np.float16)
tensor_description = TensorDesc(precision="FP16", dims=(1, 1, 512, 512), layout='NCHW')
input_blob = Blob(tensor_description, input_data)
input_blob_name = next(iter(inference_request.input_blobs))
inference_request.set_blob(blob_name=input_blob_name, blob=input_blob)
inference_request.infer()
output_blob_name = next(iter(inference_request.output_blobs))
output = inference_request.output_blobs[output_blob_name].buffer
print(output)
它适用于我的其他 3 个频道项目。
但是这个 1 通道模型出现了错误。
Traceback (most recent call last):
File "C:\Users\Hsien\Desktop\seg_test.py", line 20, in <module>
input_blob = Blob(tensor_description, input_data)
File "ie_api.pyx", line 214, in openvino.inference_engine.ie_api.Blob.__cinit__
AttributeError: Number of elements in provided numpy array 786432 and required by TensorDesc 262144 are not equal
所以这意味着 OpenVINO 无论如何都不能支持灰度模型？
目前，Open Model Zoo 演示和示例（segmentation_demo.py、classification_sample_async.py）仅支持 3 个输入通道，按 BGR 通道顺序排列。
Open Model Zoo 演示和模型的 1 个输入通道可能会在未来的版本中得到支持和启用。
如果模型针对 3 通道 RGB/BGR 输入进行训练，则您需要将 3 通道内容输入模型。
您是否尝试过将 1-channel-input 转换为 3-channel-input，即将“纯灰度图像”转换为彩色图像（“mycolor=cv2.cvtColor(灰色，cv2.COLOR_GRAY2RGB)")?或使用例如NUMPY 添加额外的 channels/dimensions?
你有“足够”的灰度图像来重新训练模型吗？
您可能想看看仍然开放的 Pull/MergeRequest“https://github.com/openvinotoolkit/training_extensions/pull/546” 乳房超声（模拟）培训。
一般不能指望用自己的模型替换模型后demo还能正常运行，因为不同的模型对输入格式布局的要求不同，pre/postprocessing。
对于这个 AnamNet，我首先通过向 https://github.com/NaveenPaluru/Segmentation-COVID-19/blob/master/Sample%20Testing/sample_test.ipynb 添加以下行将模型导出为 onnx 格式（可能你必须确保模型在 CPU 而不是 GPU 上，我不是非常肯定）：
torch.onnx.export(net, (inp1, ), 'model.onnx')
然后我将它加载到 openvino 和 运行 推理（sampledata.mat 来自 https://github.com/NaveenPaluru/Segmentation-COVID-19/）：
import scipy.io as io
import openvino.runtime as ov
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
core = ov.Core()
model = core.read_model('model.onnx')
device = 'CPU'
model_cpu = core.compile_model(model, device)
k0 = model_cpu.output(0)
data = io.loadmat('sampledata.mat')
fig, axs = plt.subplots(1, 4, figsize=(50, 50))
for i, name in enumerate(['im1','im2','im3','im4']):
inp = data[name]
inp = np.reshape(inp,(512,512,1))
inp = transforms.ToTensor()(inp).unsqueeze(dim=0)
out = model_cpu.infer_new_request([inp])[k0]
axs[i].imshow(out)
plt.show()
它重现了与我在
https://github.com/NaveenPaluru/Segmentation-COVID-19/blob/master/Sample%20Testing/sample_test.ipynb
至于为什么您的代码（基于 OpennVINO API 1.0）不起作用，从您显示的错误日志中可以看出：
AttributeError: Number of elements in provided numpy array 786432 and required by TensorDesc 262144 are not equal
我猜 OpenCV 从 786432=512x512x3 开始仍然生成 3 通道，并且模型只接受 262144=512x512 输入，因此您可以按照 this link 中描述的说明确保在输入之前生成灰度数组OpenVINO.
我尝试使用 COVID-19 CT segmentation dataset 和 NCS2 进行 CT 切片分割。
我引用 this GitHub 并使用其 pth 文件转换为 ONNX。
我用python mo_onnx.py --input_model model.onnx --input_shape [1,1,512,512] --data_type FP16生成IR文件。
我预期的推理结果如下图所示。
我在openvino_2021.4.689版本中试过segmentation_demo.py，但是报错如下。
[ INFO ] Initializing Inference Engine...
[ INFO ] Loading network...
[ INFO ] Reading network from IR...
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 270, in <module>
sys.exit(main() or 0)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 176, in main
model, visualizer = get_model(ie, args)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\inference_engine\demos\segmentation_demo\python\segmentation_demo.py", line 160, in get_model
return SegmentationModel(ie, args.model), SegmentationVisualizer(args.colors)
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\open_model_zoo\demos\common\python\models\segmentation.py", line 27, in __init__
self.input_blob_name = self.prepare_inputs()
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\deployment_tools\open_model_zoo\demos\common\python\models\segmentation.py", line 43, in prepare_inputs
raise RuntimeError("3-channel 4-dimensional model's input is expected")
RuntimeError: 3-channel 4-dimensional model's input is expected
然后，我也尝试运行classification_sample_async.py，遇到下面类似的问题。
[ INFO ] Creating Inference Engine
[ INFO ] Reading the network: C:\Users\Hsien\Desktop\Segmentation-COVID-19-master\Sample_Testing\model.xml
[ INFO ] Configuring input and output blobs
[ INFO ] Loading the model to the plugin
[ INFO ] Starting inference in asynchronous mode
Traceback (most recent call last):
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\inference_engine\samples\python\classification_sample_async\classification_sample_async.py", line 168, in <module>
sys.exit(main())
File "C:\Program Files (x86)\Intel\openvino_2021.4.689\inference_engine\samples\python\classification_sample_async\classification_sample_async.py", line 111, in main
exec_net.requests[i].async_infer({input_blob: input_data[i]})
File "ie_api.pyx", line 1222, in openvino.inference_engine.ie_api.InferRequest.async_infer
File "ie_api.pyx", line 1224, in openvino.inference_engine.ie_api.InferRequest.async_infer
File "ie_api.pyx", line 1363, in openvino.inference_engine.ie_api.InferRequest._fill_inputs
ValueError: could not broadcast input array from shape (1,3,512,512) into shape (1,1,512,512)
两条消息都可以看出，演示代码不支持灰度图
但是如何正确处理灰度CT切片分割呢？
我还提到了 210-ct-scan-live-inference.ipynb 和 110-ct-segmentation-quantize.ipynb 在 openvino_notebooks 上 GitHub 但没有成功。
（我的环境是Windows11，版本是openvino_2021.4.689。）
更新新的测试结果。
我在下面开发了一个简单的推理代码。
from openvino.inference_engine import IECore, Blob, TensorDesc
import cv2
import numpy as np
IMG_PATH = "20.jpg"
XML_PATH = "model.xml"
BIN_PATH = "model.bin"
ie_core_handler = IECore()
network = ie_core_handler.read_network(model=XML_PATH, weights=BIN_PATH)
input_blob_name = next(iter(network.input_info))
network.input_info[input_blob_name].precision = 'FP16'
executable_network = ie_core_handler.load_network(network, device_name='MYRIAD', num_requests=1)
inference_request = executable_network.requests[0]
image = cv2.imread(IMG_PATH)
re_img = cv2.resize(src=image, dsize=(512, 512))
input_data = np.expand_dims(np.transpose(re_img, (2, 0, 1)), 0).astype(np.float16)
tensor_description = TensorDesc(precision="FP16", dims=(1, 1, 512, 512), layout='NCHW')
input_blob = Blob(tensor_description, input_data)
input_blob_name = next(iter(inference_request.input_blobs))
inference_request.set_blob(blob_name=input_blob_name, blob=input_blob)
inference_request.infer()
output_blob_name = next(iter(inference_request.output_blobs))
output = inference_request.output_blobs[output_blob_name].buffer
print(output)
它适用于我的其他 3 个频道项目。
但是这个 1 通道模型出现了错误。
Traceback (most recent call last):
File "C:\Users\Hsien\Desktop\seg_test.py", line 20, in <module>
input_blob = Blob(tensor_description, input_data)
File "ie_api.pyx", line 214, in openvino.inference_engine.ie_api.Blob.__cinit__
AttributeError: Number of elements in provided numpy array 786432 and required by TensorDesc 262144 are not equal
所以这意味着 OpenVINO 无论如何都不能支持灰度模型？
目前，Open Model Zoo 演示和示例（segmentation_demo.py、classification_sample_async.py）仅支持 3 个输入通道，按 BGR 通道顺序排列。
Open Model Zoo 演示和模型的 1 个输入通道可能会在未来的版本中得到支持和启用。
如果模型针对 3 通道 RGB/BGR 输入进行训练，则您需要将 3 通道内容输入模型。
您是否尝试过将 1-channel-input 转换为 3-channel-input，即将“纯灰度图像”转换为彩色图像（“mycolor=cv2.cvtColor(灰色，cv2.COLOR_GRAY2RGB)")?或使用例如NUMPY 添加额外的 channels/dimensions?
你有“足够”的灰度图像来重新训练模型吗？
您可能想看看仍然开放的 Pull/MergeRequest“https://github.com/openvinotoolkit/training_extensions/pull/546” 乳房超声（模拟）培训。
一般不能指望用自己的模型替换模型后demo还能正常运行，因为不同的模型对输入格式布局的要求不同，pre/postprocessing。
对于这个 AnamNet，我首先通过向 https://github.com/NaveenPaluru/Segmentation-COVID-19/blob/master/Sample%20Testing/sample_test.ipynb 添加以下行将模型导出为 onnx 格式（可能你必须确保模型在 CPU 而不是 GPU 上，我不是非常肯定）：
torch.onnx.export(net, (inp1, ), 'model.onnx')
然后我将它加载到 openvino 和 运行 推理（sampledata.mat 来自 https://github.com/NaveenPaluru/Segmentation-COVID-19/）：
import scipy.io as io
import openvino.runtime as ov
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
core = ov.Core()
model = core.read_model('model.onnx')
device = 'CPU'
model_cpu = core.compile_model(model, device)
k0 = model_cpu.output(0)
data = io.loadmat('sampledata.mat')
fig, axs = plt.subplots(1, 4, figsize=(50, 50))
for i, name in enumerate(['im1','im2','im3','im4']):
inp = data[name]
inp = np.reshape(inp,(512,512,1))
inp = transforms.ToTensor()(inp).unsqueeze(dim=0)
out = model_cpu.infer_new_request([inp])[k0]
axs[i].imshow(out)
plt.show()
它重现了与我在 https://github.com/NaveenPaluru/Segmentation-COVID-19/blob/master/Sample%20Testing/sample_test.ipynb
至于为什么您的代码（基于 OpennVINO API 1.0）不起作用，从您显示的错误日志中可以看出：
AttributeError: Number of elements in provided numpy array 786432 and required by TensorDesc 262144 are not equal
我猜 OpenCV 从 786432=512x512x3 开始仍然生成 3 通道，并且模型只接受 262144=512x512 输入，因此您可以按照 this link 中描述的说明确保在输入之前生成灰度数组OpenVINO.