Tensorflow Object Detection 在同一模型检查点的两次评估中得到不同的结果
Tensorflow Object Detection got different results in two evaluations on the same model checkpoint
我是 tensorflow 的菜鸟,目前从事对象检测 API。
我选择了 ssd_resnet50_fpn 开始并从 tensorflow model zoo 下载了预训练模型,用我自己的数据集进行迁移学习,只有 1 class(人)。训练配置是在 pipeline.config 中定义的,它是从预训练模型的同一包中的那个修改而来的,我用遗留的 train.py 脚本训练了模型。
训练过程很好,损失按预期减少,并且我在进行评估时导出了带有边界框的评估图像(通过旧版 eval.py)。推理很好,导出的图像按预期工作。
但是,我发现了一些奇怪的事情,我在同一个模型检查点上得到了不同的评估结果。当我用相同的参数执行两次 eval.py 时,我发现在同一张图像上推断出的边界框是不同的。
这是eval.py(coco_detection_metrics)
的评价结果
由于我对mAP不是很了解,但两次结果略有不同。
这里是评测时导出的一张图,左边是第一次评测,右边是第二次评测
Exported Image during evaluation
似乎模型权重在推理过程中发生了变化,
我怎样才能找出问题所在?有没有我错过的配置?
我正在使用带有 python 3.5.2 的 tensorflow 1.10.1 和 https://github.com/tensorflow/models 中的克隆对象检测 API,没有任何变化。
这是我的 pipeline.config:
model {
ssd {
num_classes: 1
image_resizer {
fixed_shape_resizer {
height: 640
width: 640
}
}
feature_extractor {
type: "ssd_resnet50_v1_fpn"
depth_multiplier: 1.0
min_depth: 16
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 0.000399999989895
}
}
initializer {
truncated_normal_initializer {
mean: 0.0
stddev: 0.0299999993294
}
}
activation: RELU_6
batch_norm {
decay: 0.996999979019
scale: true
epsilon: 0.0010000000475
}
}
override_base_feature_extractor_hyperparams: true
}
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
use_matmul_gather: true
}
}
similarity_calculator {
iou_similarity {
}
}
box_predictor {
weight_shared_convolutional_box_predictor {
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 0.000399999989895
}
}
initializer {
random_normal_initializer {
mean: 0.0
stddev: 0.00999999977648
}
}
activation: RELU_6
batch_norm {
decay: 0.996999979019
scale: true
epsilon: 0.0010000000475
}
}
use_dropout: true
dropout_keep_probability: 0.7
depth: 256
num_layers_before_predictor: 4
kernel_size: 3
class_prediction_bias_init: -4.59999990463
}
}
anchor_generator {
multiscale_anchor_generator {
min_level: 3
max_level: 7
anchor_scale: 4.0
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
scales_per_octave: 2
}
}
post_processing {
batch_non_max_suppression {
score_threshold: 0.300000011921
iou_threshold: 0.600000023842
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
normalize_loss_by_num_matches: true
loss {
localization_loss {
weighted_smooth_l1 {
}
}
classification_loss {
weighted_sigmoid_focal {
gamma: 2.0
alpha: 0.25
}
}
classification_weight: 1.0
localization_weight: 1.0
}
encode_background_as_zeros: true
normalize_loc_loss_by_codesize: true
inplace_batchnorm_update: true
freeze_batchnorm: false
}
}
train_config {
batch_size: 8
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
random_crop_image {
min_object_covered: 0.0
min_aspect_ratio: 0.75
max_aspect_ratio: 3.0
min_area: 0.75
max_area: 1.0
overlap_thresh: 0.0
}
}
sync_replicas: false
optimizer {
adam_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.0001
decay_steps: 5000
decay_factor: 0.9
}
}
}
use_moving_average: false
}
fine_tune_checkpoint: "/tf-object-detection-training/models/ssd_resnet50/saved/model.ckpt-652123"
num_steps: 2000000
from_detection_checkpoint: true
load_all_detection_checkpoint_vars: true
startup_delay_steps: 0.0
replicas_to_aggregate: 8
max_number_of_boxes: 100
unpad_groundtruth_tensors: false
}
train_input_reader {
label_map_path: "/tf-object-detection-training/dataset_VOC/label.pbtxt"
tf_record_input_reader {
input_path: "/tf-object-detection-training/dataset_VOC/person_train.record-?????-of-00010"
}
}
eval_config {
num_examples: 10000
num_visualizations: 100
eval_interval_secs: 60
metrics_set: "coco_detection_metrics"
use_moving_averages: false
min_score_threshold: 0.5
retain_original_images: false
keep_image_id_for_visualization_export: true
visualization_export_dir: "/tf-object-detection-training/models/ssd_resnet50/eval_detections/"
}
eval_input_reader {
label_map_path: "/tf-object-detection-training/dataset_VOC/label.pbtxt"
shuffle: false
num_readers: 1
tf_record_input_reader {
input_path: "/tf-object-detection-training/dataset_VOC/person_val.record-?????-of-00010"
}
}
感谢任何建议
查了半天代码,发现是在pipeline.config中设置了'use_dropout'标志位。
似乎在进行推理时没有删除 dropout 函数,因此 eval.py 和 frozen_inference_graph 都应用了 dropout 函数并进行随机推理。
要解决此问题,只需从 pipeline.config 中删除 'use_dropout' 即可解决此问题。
我是 tensorflow 的菜鸟,目前从事对象检测 API。
我选择了 ssd_resnet50_fpn 开始并从 tensorflow model zoo 下载了预训练模型,用我自己的数据集进行迁移学习,只有 1 class(人)。训练配置是在 pipeline.config 中定义的,它是从预训练模型的同一包中的那个修改而来的,我用遗留的 train.py 脚本训练了模型。
训练过程很好,损失按预期减少,并且我在进行评估时导出了带有边界框的评估图像(通过旧版 eval.py)。推理很好,导出的图像按预期工作。
但是,我发现了一些奇怪的事情,我在同一个模型检查点上得到了不同的评估结果。当我用相同的参数执行两次 eval.py 时,我发现在同一张图像上推断出的边界框是不同的。
这是eval.py(coco_detection_metrics)
的评价结果由于我对mAP不是很了解,但两次结果略有不同。
这里是评测时导出的一张图,左边是第一次评测,右边是第二次评测
Exported Image during evaluation
似乎模型权重在推理过程中发生了变化, 我怎样才能找出问题所在?有没有我错过的配置?
我正在使用带有 python 3.5.2 的 tensorflow 1.10.1 和 https://github.com/tensorflow/models 中的克隆对象检测 API,没有任何变化。
这是我的 pipeline.config:
model {
ssd {
num_classes: 1
image_resizer {
fixed_shape_resizer {
height: 640
width: 640
}
}
feature_extractor {
type: "ssd_resnet50_v1_fpn"
depth_multiplier: 1.0
min_depth: 16
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 0.000399999989895
}
}
initializer {
truncated_normal_initializer {
mean: 0.0
stddev: 0.0299999993294
}
}
activation: RELU_6
batch_norm {
decay: 0.996999979019
scale: true
epsilon: 0.0010000000475
}
}
override_base_feature_extractor_hyperparams: true
}
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
use_matmul_gather: true
}
}
similarity_calculator {
iou_similarity {
}
}
box_predictor {
weight_shared_convolutional_box_predictor {
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 0.000399999989895
}
}
initializer {
random_normal_initializer {
mean: 0.0
stddev: 0.00999999977648
}
}
activation: RELU_6
batch_norm {
decay: 0.996999979019
scale: true
epsilon: 0.0010000000475
}
}
use_dropout: true
dropout_keep_probability: 0.7
depth: 256
num_layers_before_predictor: 4
kernel_size: 3
class_prediction_bias_init: -4.59999990463
}
}
anchor_generator {
multiscale_anchor_generator {
min_level: 3
max_level: 7
anchor_scale: 4.0
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
scales_per_octave: 2
}
}
post_processing {
batch_non_max_suppression {
score_threshold: 0.300000011921
iou_threshold: 0.600000023842
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
normalize_loss_by_num_matches: true
loss {
localization_loss {
weighted_smooth_l1 {
}
}
classification_loss {
weighted_sigmoid_focal {
gamma: 2.0
alpha: 0.25
}
}
classification_weight: 1.0
localization_weight: 1.0
}
encode_background_as_zeros: true
normalize_loc_loss_by_codesize: true
inplace_batchnorm_update: true
freeze_batchnorm: false
}
}
train_config {
batch_size: 8
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
random_crop_image {
min_object_covered: 0.0
min_aspect_ratio: 0.75
max_aspect_ratio: 3.0
min_area: 0.75
max_area: 1.0
overlap_thresh: 0.0
}
}
sync_replicas: false
optimizer {
adam_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.0001
decay_steps: 5000
decay_factor: 0.9
}
}
}
use_moving_average: false
}
fine_tune_checkpoint: "/tf-object-detection-training/models/ssd_resnet50/saved/model.ckpt-652123"
num_steps: 2000000
from_detection_checkpoint: true
load_all_detection_checkpoint_vars: true
startup_delay_steps: 0.0
replicas_to_aggregate: 8
max_number_of_boxes: 100
unpad_groundtruth_tensors: false
}
train_input_reader {
label_map_path: "/tf-object-detection-training/dataset_VOC/label.pbtxt"
tf_record_input_reader {
input_path: "/tf-object-detection-training/dataset_VOC/person_train.record-?????-of-00010"
}
}
eval_config {
num_examples: 10000
num_visualizations: 100
eval_interval_secs: 60
metrics_set: "coco_detection_metrics"
use_moving_averages: false
min_score_threshold: 0.5
retain_original_images: false
keep_image_id_for_visualization_export: true
visualization_export_dir: "/tf-object-detection-training/models/ssd_resnet50/eval_detections/"
}
eval_input_reader {
label_map_path: "/tf-object-detection-training/dataset_VOC/label.pbtxt"
shuffle: false
num_readers: 1
tf_record_input_reader {
input_path: "/tf-object-detection-training/dataset_VOC/person_val.record-?????-of-00010"
}
}
感谢任何建议
查了半天代码,发现是在pipeline.config中设置了'use_dropout'标志位。 似乎在进行推理时没有删除 dropout 函数,因此 eval.py 和 frozen_inference_graph 都应用了 dropout 函数并进行随机推理。
要解决此问题,只需从 pipeline.config 中删除 'use_dropout' 即可解决此问题。