使用 CNN 训练海量数据
Training Huge Data with CNN
作为机器学习的新手，我尝试搜索了很多用于训练分类器的方法以及如何使数据可读。到目前为止，我所知道的是，标签对于分类器来说是最重要的，这是显而易见的。我的问题是，我在其他文件夹中有超过 300000 个图像和标签的庞大数据集，解释了每个图像的侧面和边界框，我还有其他数据信息，这些信息在其他文件夹中，而不是在标签文件夹中，而是在 misc 文件夹中.mat 文件和每个 .mat 文件都包含图像中汽车的品牌或型号名称。因为，到目前为止，我正在以 numpy 格式转换图像和标签数据，并将它们附加在 training_data 中，我应该如何处理 misc 数据以便它也可以使用标签和图像数据进行训练。
非常感谢您的回答。
我在下面对文件夹进行更多的解释，让大家对数据有更多的了解。我只需要理论上的答案步骤，如果你们能做到的话。
数据说明
文件夹和文件说明如下：
-image:
Stores all full car images in the path format 'make_id/model_id/released_year/image_name.jpg'.
-label:
Stores all labels to the full car images in the path format 'make_id/model_id/released_year/image_name.txt'. Each label file has three lines. The first line is a number which is the viewpoint annotation (-1 - uncertain, 1 - front, 2 - rear, 3 - side, 4 - front-side, 5 - rear-side). The second line is the number of the bounding boxes, which is all '1' in the current release. The third line is the coordinates of the bounding box in the format 'x1 y1 x2 y2' in pixels, where 1 <= x1 < x2 <= image_width, and 1 <= y1 < y2 <= image_height.
-misc:
-attributes.txt:
Each line is the attribute annotation for one model which is in the format 'model_id maximum_speed displacement door_number seat_number type'. For car types, a number from 1~12 corresponds to a specific type, which is described in 'car_type.mat'. Unavailable attributes are denoted by '0' or '0.0'.
-make_model_name.mat
Cell array 'make_names' provides the projections from 'make_id' to make names, and cell array 'model_names' provides the projections from 'model_id' to model names.
-part:
Stores all part images in the path format 'make_id/model_id/released_year/part_id/image_name.jpg'. The correspondance of 'part_id' and part names are: 1 - headlight, 2 - taillight, 3 - fog light, 4 - air intake, 5 - console, 6 - steering wheel, 7 - dashboard, and 8 - gear lever.
-train_test_split:
This folder generally provides all the train/test subsets used in the paper.
-classification
Stores the train/test lists for the classification task with full car images in the paper.
-part:
Stores the train/test lists for the classification task with car part in the paper.
-verification:
'verification_train.txt' is the image list for training the verification models which is also for testing attribute prediction. 'verification_pairs_easy.txt', 'verification_pairs_medium.txt', and 'verification_pairs_hard.txt' are the three sets with different difficulties for testing car verification models. Each line of 'verification_pairs_XXX.txt' is in the format of 'path_to_image_1 path_to_image_2 label' where label is '1' for positive pairs and is '0' for negative pairs.
谢谢。
你可以做的是把杂项数据变成数字。
假设您有 3 种不同型号的汽车：Ferrari、Tesla 和 Lambo。您可以定义如果汽车是Ferrari，它的编号是0。如果是Tesla，它的编号是1，如果是Lambo，它的编号是2。现在，当您加载标签时，也加载 misc 文件夹数据并执行上面定义的交换：Ferrari = 0, Tesla = 1, Lambo = 2 现在您可以将其附加到标签向量并教网络预测 0，1 或 2 表示汽车型号。当网络做出预测后，您可以断言预测是否为真（例如，如果 NN 预测 1，则意味着图像中的汽车是 Tesla）。
您可以将此方法应用于 misc 文件夹中的任何其他非数字特征。这就是所谓的 embedding - 将非数值转换为数值。
作为机器学习的新手，我尝试搜索了很多用于训练分类器的方法以及如何使数据可读。到目前为止，我所知道的是，标签对于分类器来说是最重要的，这是显而易见的。我的问题是，我在其他文件夹中有超过 300000 个图像和标签的庞大数据集，解释了每个图像的侧面和边界框，我还有其他数据信息，这些信息在其他文件夹中，而不是在标签文件夹中，而是在 misc 文件夹中.mat 文件和每个 .mat 文件都包含图像中汽车的品牌或型号名称。因为，到目前为止，我正在以 numpy 格式转换图像和标签数据，并将它们附加在 training_data 中，我应该如何处理 misc 数据以便它也可以使用标签和图像数据进行训练。
非常感谢您的回答。
我在下面对文件夹进行更多的解释，让大家对数据有更多的了解。我只需要理论上的答案步骤，如果你们能做到的话。
数据说明
文件夹和文件说明如下：
-image:
Stores all full car images in the path format 'make_id/model_id/released_year/image_name.jpg'.
-label:
Stores all labels to the full car images in the path format 'make_id/model_id/released_year/image_name.txt'. Each label file has three lines. The first line is a number which is the viewpoint annotation (-1 - uncertain, 1 - front, 2 - rear, 3 - side, 4 - front-side, 5 - rear-side). The second line is the number of the bounding boxes, which is all '1' in the current release. The third line is the coordinates of the bounding box in the format 'x1 y1 x2 y2' in pixels, where 1 <= x1 < x2 <= image_width, and 1 <= y1 < y2 <= image_height.
-misc:
-attributes.txt:
Each line is the attribute annotation for one model which is in the format 'model_id maximum_speed displacement door_number seat_number type'. For car types, a number from 1~12 corresponds to a specific type, which is described in 'car_type.mat'. Unavailable attributes are denoted by '0' or '0.0'.
-make_model_name.mat
Cell array 'make_names' provides the projections from 'make_id' to make names, and cell array 'model_names' provides the projections from 'model_id' to model names.
-part:
Stores all part images in the path format 'make_id/model_id/released_year/part_id/image_name.jpg'. The correspondance of 'part_id' and part names are: 1 - headlight, 2 - taillight, 3 - fog light, 4 - air intake, 5 - console, 6 - steering wheel, 7 - dashboard, and 8 - gear lever.
-train_test_split:
This folder generally provides all the train/test subsets used in the paper.
-classification
Stores the train/test lists for the classification task with full car images in the paper.
-part:
Stores the train/test lists for the classification task with car part in the paper.
-verification:
'verification_train.txt' is the image list for training the verification models which is also for testing attribute prediction. 'verification_pairs_easy.txt', 'verification_pairs_medium.txt', and 'verification_pairs_hard.txt' are the three sets with different difficulties for testing car verification models. Each line of 'verification_pairs_XXX.txt' is in the format of 'path_to_image_1 path_to_image_2 label' where label is '1' for positive pairs and is '0' for negative pairs.
谢谢。
你可以做的是把杂项数据变成数字。
假设您有 3 种不同型号的汽车：Ferrari、Tesla 和 Lambo。您可以定义如果汽车是Ferrari，它的编号是0。如果是Tesla，它的编号是1，如果是Lambo，它的编号是2。现在，当您加载标签时，也加载 misc 文件夹数据并执行上面定义的交换：Ferrari = 0, Tesla = 1, Lambo = 2 现在您可以将其附加到标签向量并教网络预测 0，1 或 2 表示汽车型号。当网络做出预测后，您可以断言预测是否为真（例如，如果 NN 预测 1，则意味着图像中的汽车是 Tesla）。
您可以将此方法应用于 misc 文件夹中的任何其他非数字特征。这就是所谓的 embedding - 将非数值转换为数值。