Detectron2 - 在对象检测的阈值处提取区域特征
Detectron2 - Extract region features at a threshold for object detection
我正在尝试使用 detectron2 framework. I will be using these features later in my pipeline (similar to: VilBert section 3.1 Training ViLBERT) So far I have trained a Mask R-CNN with this config 提取 class 检测高于某个阈值的区域特征，并在某些自定义数据上对其进行微调。它表现良好。我想做的是从我训练的模型中为生成的边界框提取特征。
编辑：我查看了关闭我的 post 的用户所写的内容并尝试对其进行完善。尽管 reader 需要关于我在做什么的上下文。如果您对我如何改进问题有任何想法，或者如果您对如何做我正在尝试做的事情有一些见解，欢迎您提供反馈！
我有一个问题：
- 为什么我只得到一个 预测实例，但是当我看
在 预测 CLS 分数 中，有超过 1 个通过了
阈值？
我相信这是生成 ROI 特征的正确方法：
images = ImageList.from_tensors(lst[:1], size_divisibility=32).to("cuda") # preprocessed input tensor
#setup config
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class (pnumonia)
#Just run these lines if you have the trained model im memory
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set the testing threshold for this model
#build model
model = build_model(cfg)
DetectionCheckpointer(model).load("output/model_final.pth")
model.eval()#make sure its in eval mode
#run model
with torch.no_grad():
features = model.backbone(images.tensor.float())
proposals, _ = model.proposal_generator(images, features)
instances = model.roi_heads._forward_box(features, proposals)
然后
pred_boxes = [x.pred_boxes for x in instances]
rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], pred_boxes)
这应该是我的 ROI 特征。
我非常困惑的是，我可以使用建议和 proposal_boxes 及其 class 分数来获得该图像的前 n 个特征，而不是使用推理产生的边界框.很酷，所以我尝试了以下方法：
proposal_boxes = [x.proposal_boxes for x in proposals]
proposal_rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], proposal_boxes)
#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
box_features = model.roi_heads.box_head(proposal_rois)
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, losses = model.roi_heads.box_predictor.inference(predictions, proposals)
我应该在哪里获得我的提案框功能及其 cls in my predictions object。检查此 predictions 对象，我看到每个框的分数：
预测对象中的 CLS 分数
(tensor([[ 0.6308, -0.4926],
[-1.6662, 1.5430],
[-0.2080, 0.4856],
...,
[-6.9698, 6.6695],
[-5.6361, 5.4046],
[-4.4918, 4.3899]], device='cuda:0', grad_fn=<AddmmBackward>),
在 softmaxing 并将这些 cls 分数放入数据框中并将阈值设置为 0.6 之后，我得到：
pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
pred_df[pred_df[0] > 0.6]
0 1
0 0.754618 0.245382
6 0.686816 0.313184
38 0.722627 0.277373
并且在我的预测对象中我得到相同的最高分，但只有 1 个实例而不是 2 个（我设置 cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7）：
预测实例：
[Instances(num_instances=1, image_height=800, image_width=800, fields=[pred_boxes: Boxes(tensor([[548.5992, 341.7193, 756.9728, 438.0507]], device='cuda:0',
grad_fn=<IndexBackward>)), scores: tensor([0.7546], device='cuda:0', grad_fn=<IndexBackward>), pred_classes: tensor([0], device='cuda:0')])]
预测还包含 Tensor：Nx4 或 Nx(Kx4) 边界框回归增量。 我不太清楚它们的作用和外观：
预测对象中的边界框回归增量
tensor([[ 0.2502, 0.2461, -0.4559, -0.3304],
[-0.1359, -0.1563, -0.2821, 0.0557],
[ 0.7802, 0.5719, -1.0790, -1.3001],
...,
[-0.8594, 0.0632, 0.2024, -0.6000],
[-0.2020, -3.3195, 0.6745, 0.5456],
[-0.5542, 1.1727, 1.9679, -2.3912]], device='cuda:0',
grad_fn=<AddmmBackward>)
还有一点奇怪的是，我的建议框和我的预测框不同但相似：
提案边界框
[Boxes(tensor([[532.9427, 335.8969, 761.2068, 438.8086],#this box vs the instance box
[102.7041, 352.5067, 329.4510, 440.7240],
[499.2719, 317.9529, 764.1958, 448.1386],
...,
[ 25.2890, 379.3329, 28.6030, 429.9694],
[127.1215, 392.6055, 328.6081, 489.0793],
[164.5633, 275.6021, 295.0134, 462.7395]], device='cuda:0'))]
你快到了。查看 roi_heads.box_predictor.inference() 你会发现它并不是简单地对 box candidates 的分数进行排序。首先，它应用框增量来重新调整建议框。然后它计算非最大抑制以删除非重叠框（同时还应用其他超设置，例如分数阈值）。最后，它根据分数对 top-k 框进行排名。这可能解释了为什么您的方法产生相同的框分数但输出框及其坐标的数量不同。
回到你原来的问题，这里是在一次推理过程中提取建议框的特征的方法：
image = cv2.imread('my_image.jpg')
height, width = image.shape[:2]
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))
inputs = [{"image": image, "height": height, "width": width}]
with torch.no_grad():
images = model.preprocess_image(inputs) # don't forget to preprocess
features = model.backbone(images.tensor) # set of cnn features
proposals, _ = model.proposal_generator(images, features, None) # RPN
features_ = [features[f] for f in model.roi_heads.box_in_features]
box_features = model.roi_heads.box_pooler(features_, [x.proposal_boxes for x in proposals])
box_features = model.roi_heads.box_head(box_features) # features of all 1k candidates
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, pred_inds = model.roi_heads.box_predictor.inference(predictions, proposals)
pred_instances = model.roi_heads.forward_with_given_boxes(features, pred_instances)
# output boxes, masks, scores, etc
pred_instances = model._postprocess(pred_instances, inputs, images.image_sizes) # scale box to orig size
# features of the proposed boxes
feats = box_features[pred_inds]
我正在尝试使用 detectron2 framework. I will be using these features later in my pipeline (similar to: VilBert section 3.1 Training ViLBERT) So far I have trained a Mask R-CNN with this config 提取 class 检测高于某个阈值的区域特征，并在某些自定义数据上对其进行微调。它表现良好。我想做的是从我训练的模型中为生成的边界框提取特征。
编辑：我查看了关闭我的 post 的用户所写的内容并尝试对其进行完善。尽管 reader 需要关于我在做什么的上下文。如果您对我如何改进问题有任何想法，或者如果您对如何做我正在尝试做的事情有一些见解，欢迎您提供反馈！
我有一个问题：
- 为什么我只得到一个 预测实例，但是当我看 在 预测 CLS 分数 中，有超过 1 个通过了 阈值？
我相信这是生成 ROI 特征的正确方法：
images = ImageList.from_tensors(lst[:1], size_divisibility=32).to("cuda") # preprocessed input tensor
#setup config
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class (pnumonia)
#Just run these lines if you have the trained model im memory
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set the testing threshold for this model
#build model
model = build_model(cfg)
DetectionCheckpointer(model).load("output/model_final.pth")
model.eval()#make sure its in eval mode
#run model
with torch.no_grad():
features = model.backbone(images.tensor.float())
proposals, _ = model.proposal_generator(images, features)
instances = model.roi_heads._forward_box(features, proposals)
然后
pred_boxes = [x.pred_boxes for x in instances]
rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], pred_boxes)
这应该是我的 ROI 特征。
我非常困惑的是，我可以使用建议和 proposal_boxes 及其 class 分数来获得该图像的前 n 个特征，而不是使用推理产生的边界框.很酷，所以我尝试了以下方法：
proposal_boxes = [x.proposal_boxes for x in proposals]
proposal_rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], proposal_boxes)
#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
box_features = model.roi_heads.box_head(proposal_rois)
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, losses = model.roi_heads.box_predictor.inference(predictions, proposals)
我应该在哪里获得我的提案框功能及其 cls in my predictions object。检查此 predictions 对象，我看到每个框的分数：
预测对象中的 CLS 分数
(tensor([[ 0.6308, -0.4926],
[-1.6662, 1.5430],
[-0.2080, 0.4856],
...,
[-6.9698, 6.6695],
[-5.6361, 5.4046],
[-4.4918, 4.3899]], device='cuda:0', grad_fn=<AddmmBackward>),
在 softmaxing 并将这些 cls 分数放入数据框中并将阈值设置为 0.6 之后，我得到：
pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
pred_df[pred_df[0] > 0.6]
0 1
0 0.754618 0.245382
6 0.686816 0.313184
38 0.722627 0.277373
并且在我的预测对象中我得到相同的最高分，但只有 1 个实例而不是 2 个（我设置 cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7）：
预测实例：
[Instances(num_instances=1, image_height=800, image_width=800, fields=[pred_boxes: Boxes(tensor([[548.5992, 341.7193, 756.9728, 438.0507]], device='cuda:0',
grad_fn=<IndexBackward>)), scores: tensor([0.7546], device='cuda:0', grad_fn=<IndexBackward>), pred_classes: tensor([0], device='cuda:0')])]
预测还包含 Tensor：Nx4 或 Nx(Kx4) 边界框回归增量。 我不太清楚它们的作用和外观：
预测对象中的边界框回归增量
tensor([[ 0.2502, 0.2461, -0.4559, -0.3304],
[-0.1359, -0.1563, -0.2821, 0.0557],
[ 0.7802, 0.5719, -1.0790, -1.3001],
...,
[-0.8594, 0.0632, 0.2024, -0.6000],
[-0.2020, -3.3195, 0.6745, 0.5456],
[-0.5542, 1.1727, 1.9679, -2.3912]], device='cuda:0',
grad_fn=<AddmmBackward>)
还有一点奇怪的是，我的建议框和我的预测框不同但相似：
提案边界框
[Boxes(tensor([[532.9427, 335.8969, 761.2068, 438.8086],#this box vs the instance box
[102.7041, 352.5067, 329.4510, 440.7240],
[499.2719, 317.9529, 764.1958, 448.1386],
...,
[ 25.2890, 379.3329, 28.6030, 429.9694],
[127.1215, 392.6055, 328.6081, 489.0793],
[164.5633, 275.6021, 295.0134, 462.7395]], device='cuda:0'))]
你快到了。查看 roi_heads.box_predictor.inference() 你会发现它并不是简单地对 box candidates 的分数进行排序。首先，它应用框增量来重新调整建议框。然后它计算非最大抑制以删除非重叠框（同时还应用其他超设置，例如分数阈值）。最后，它根据分数对 top-k 框进行排名。这可能解释了为什么您的方法产生相同的框分数但输出框及其坐标的数量不同。
回到你原来的问题，这里是在一次推理过程中提取建议框的特征的方法：
image = cv2.imread('my_image.jpg')
height, width = image.shape[:2]
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))
inputs = [{"image": image, "height": height, "width": width}]
with torch.no_grad():
images = model.preprocess_image(inputs) # don't forget to preprocess
features = model.backbone(images.tensor) # set of cnn features
proposals, _ = model.proposal_generator(images, features, None) # RPN
features_ = [features[f] for f in model.roi_heads.box_in_features]
box_features = model.roi_heads.box_pooler(features_, [x.proposal_boxes for x in proposals])
box_features = model.roi_heads.box_head(box_features) # features of all 1k candidates
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, pred_inds = model.roi_heads.box_predictor.inference(predictions, proposals)
pred_instances = model.roi_heads.forward_with_given_boxes(features, pred_instances)
# output boxes, masks, scores, etc
pred_instances = model._postprocess(pred_instances, inputs, images.image_sizes) # scale box to orig size
# features of the proposed boxes
feats = box_features[pred_inds]