2.1 Data Description

The dataset provides images of plastic toy bricks with surface damage caused by a hammer as objects to inspect. While the bricks occur in multiple colors and sizes, the labels are provided binary with valid bricks and defective ones having damages on their surfaces. The dataset consists of 1500 images containing a total of approximately 4400 objects. Among these objects, there are roughly 2000 instances representing defects and 2400 representing valid instances. This balanced distribution of labels within the dataset serves to counteract possible biases and prevent models from learning disproportionately toward any particular class and therefore simplify the object detection task. Nevertheless, the dataset can be manipulated to introduce class imbalance by utilizing the metadata on class distribution to select a subset of the data, thereby making the task more challenging, as demonstrated in Section 3.

Each image has a corresponding label. Table 1 shows all information provided by a label. The coordinates x-center and y-center are normalized and refer to the coordinates of the center point of a bounding box, that labels an object to inspect. Width and height represent the dimensions of the bounding box in normalized pixels, where pixel values are scaled between 0 and 1, relative to the image dimensions. Lastly, the label indicates the two classes valid and defective. Figure 1 overlaps the labels of each image. Figure 1a shows x-center and y-center. The uniform distribution counteracts any specific patterns in the locations of objects. Further, Figure 1b represents the height and width of each bounding box center and indicates the dimension of an object. The linear distribution occurs due to the quadratic geometry of all plastic bricks used. Defective instances in the testing set do not appear in the training or validation set images.

Figure 1: The distribution in both figures is nearly uniform and therefore counteracts specific patterns in object locations

The correlogram in Figure 2 shows a detailed correlation of all data properties. It is a group of 2-dimensional histograms showing each axis of the data against each other axis. The correlation statistics indicate the position, width, and height of the bounding boxes of the objects. The figure indicates that the dataset properties are balanced in each label combination with no clusters visible. The distributions of single labels present approximately normal distribution. Notably, outliers are infrequent, and those present are rare points rather than data values that significantly deviate from the expected pattern.

Figure 2: The correlation of all labels to each other shows an approximately normal distribution and balance in the data

Each image is saved in JPG/.jpg format with a size ranging between 35 and 40 kilobytes. These images maintain a consistent shape of 640x640 pixels. The corresponding labels for these images are stored in a separate file in TXT/.txt format, which also includes metadata on how many instances of each class are present in the image, allowing for the creation of imbalanced subsets of the data. The file paths for both the images and the labels are specified within a file in YAML/.yaml format. As a result, all files collectively occupy a total size of 58.2 megabytes. The files are available on Zenodo and linked in Section 4: Usage Notes.

The dataset offers a wide range of possibilities for diverse tasks, including object localization, object classification, object counting, semantic segmentation, and scene understanding. However, the dataset’s provided labels and the identifiable damages on the objects make it particularly well-suited for tasks related to object detection and quality classification, specifically in identifying surface defects often encountered in manufacturing industries.

2.2 Data Collection

The data collection was done in a defined procedure. Images were captured with a microcontroller board and a compatible camera. An Arduino UNO was chosen with an OV7670 300KP VGA Camera. Arduino embedded systems are widely available and used for prototype purposes. They benefit from an active online community helping to lower development challenges [16]. Moreover, the setup includes fluorescent lighting directed toward the objects under inspection, with the camera positioned on a tripod to maintain a fixed distance from the ground, where all the objects are placed. The distance between the camera and the objects is determined by the angle each object has relative to the camera. On the software side, Python code controls the capturing process. The collection started from single objects with different colors, angles, and positions, as well as defects on some objects. Later on, multiple objects were placed in one image with the same differences described. Each defect is generated by a hammer manually and therefore individual with a varying degree of surface damage. This supports the diversity of surface damages that are labeled as defective.

The annotation of the images is based on the software Roboflow [17]. Features, polygon bounding boxes, and labels are provided with this software. Besides, Roboflow is used for auto-orient to discard common rotations by metadata and standardize pixel ordering, as well as resizing the images to a frame of 640x640 pixels from the original camera resolution of 1640x1232 pixels. This resolution of 640x640 is often suggested to facilitate the convenient use of object detection models, such as YOLOv5 [18]. Figure 3a shows an exemplary image before annotation and Figure 3b shows the same image after annotation. The purple box indicates the valid object, while the red box indicates the defective one. Table 1 shows the corresponding label information of Figure 3b. All boxes are applied comprehensively around the relevant objects, ensuring that occluded objects are always fully included. Besides, we aimed to minimize the spaces between the bounding box borders and the objects to ensure that only the relevant objects are enclosed within the box.

Figure 3: Examplary image of the dataset consisting of two objects with one valid and one defective instance

Finally, the captured images and labels are stored in Zenodo and saved with a Data Management Plan (DMP) created with RDMO [19]. The DMP includes information about metadata, data formats, as well as technical insights to enhance scientific reuse within FAIR principles. F-UJI scored the resource with a FAIRness of 75 percent.

3.1 Metrics

As the task consists of binary defect detection on objects that need to be detected first, several metrics need to be used. The object detection is measured by Intersection over Union (IoU), as suggested by literature [20]. This metric is based on the ratio of the area of intersection of two bounding boxes to the area of union of two bounding boxes as shown in the Formula 1

$IoU=\frac{Area\hspace{0.33em}of\hspace{0.33em}Intersection\hspace{0.33em}of\hspace{0.33em}two\hspace{0.33em}bounding\hspace{0.33em}boxes}{Area\hspace{0.33em}of\hspace{0.33em}Union\hspace{0.33em}of\hspace{0.33em}two\hspace{0.33em}bounding\hspace{0.33em}boxes}$ (1)

Therefore, greater IoU values signify increased overlap and an improved prediction. To eliminate redundant boxes encompassing the same object, IoU typically employs Non-Maximum Suppression. This method operates on the criterion that predictions with IoU lower than the confidence threshold are ignored, while only boxes with IoU values exceeding this threshold are retained. Here, the confidence threshold denotes the minimum score at which the model considers a prediction to be valid. Furthermore, Precision (P) and Recall (R) as classification metrics are applied to measure the accuracy of fault detection within detected objects. Generally, an image typically contains a wealth of information, including both relevant and irrelevant objects. To clarify this, P is introduced to only indicate relevant ones. It measures the proportion of correctly recognized objects out of all detected objects. R, on the other hand, measures the proportion of relevant objects that were correctly recognized by the model out of all relevant objects. The mathematical definitions of P and R are shown in Formula 2 and Formula 3. True Positive (TP) represents correct detections (IoU ≥ confidence threshold), False Positive (FP) represents a wrong detection (IoU < confidence threshold), and False Negative (FN) represents a wrong misdetection.

$Precision(P)=\frac{True\hspace{0.33em}Positive}{True\hspace{0.33em}Positive+False\hspace{0.33em}Positive}=\frac{TP}{TP+FP}$ (2)

$Recall(R)=\frac{True\hspace{0.33em}Positive}{True\hspace{0.33em}Positive+False\hspace{0.33em}Negative}=\frac{TP}{TP+FN}$ (3)

P and R offer a trade-off that it is graphically represented in the PR curve by varying the classification threshold. The area under this curve provides the average precision for each class (AP_i) for the trained model. The average of this value across all classes is referred to as the mean Average Precision (mAP), which is used to evaluate performance in object detection and quality classification in this showcase, as it combines all introduced metrics. The equation is shown in Formula 4. The equation is shown in Formula 4.

$mAP=\frac{1}{N}{\displaystyle \sum _{i=1}^{N}A}{P}_i$ (4)

N corresponds to the total number of object classes. mAP has different categories, varying in their parameter settings. We select the most common ones mAP@0.5 and mAP@0.5:0.95. mAP@0.5 is used across several benchmark challenges on datasets such as Pascal VOC or COCO. It interpolates with 101 recall points with (IoU) threshold = 0.5, which means that IoU values greater than or equal to 0.5 are considered TP, while values less than 0.5 are considered FP predictions. mAP@0.5:0.95 uses the same interpolation method as mAP@0.5, but averages the APs obtained from using ten different IoU thresholds (0.5, 0.55, …, 0.95). The introduced metrics P, R, mAP@0.5 and mAP@0.5:0.95 measure the performance of the algorithm during training and in tests after training in this showcase.

3.2 Algorithm and Training

An algorithm of YOLO series is selected as an example real-time object detection algorithm commonly used in research and industry. YOLO series object detection algorithms use a one-stage neural network to directly complete detection object localization and classification without using pre-generated region proposals [21], [22]. They are widely used for their good balance between high speed and high accuracy, easy implementation, and low-cost maintenance. YOLOv5, proposed by Jocher Glenn [18], is selected as the YOLO version after consideration of computing resources, layers of the network, model parameters, detection accuracy, inference time, deployment ability, and algorithm practicability. The specific model YOLOv5 is used for its properties of lightweight and relatively high speed. Since the size of the dataset in this showcase is relatively small and the background information is fixed, real-time detection and high accuracy can be ensured by YOLOv5s at the same time.

Training is conducted on smaller subsets of the dataset, as well as with class imbalances, to demonstrate the model’s performance to the number of images and the degree of class imbalance used for training. Three different dataset sizes, with a class imbalance in the first two, are used as shown in Table 2. To create the imbalanced datasets, images with fewer instances of the ’valid’ class were selectively removed, resulting in a final dataset with around 65% of images containing valid parts. The sizes of the training datasets are 310, 378, and 1050, respectively. The validation and testing set sizes are 16% and 20% for the 1st and 2nd datasets, and 20% and 10% of the total data for the complete dataset. The algorithm is trained 300 epochs with a batch size of 32 using YOLOv5s default hyperparameters.

3.3 Evaluation

As introduced, the results are presented with P, R, mAP@0.5 and mAP@0.5:0.95 for validation and testing set of the dataset and visualized in Table 3 and Table 4. The performance on the validation set exceeds that of the testing set, indicating overfitting during training. Overall, the performance on the testing data varies depending on dataset size and class imbalance. Regarding the entire dataset, the trained model achieves a validation mAP@0.5 of 0.995 and test a mAP@0.5 of 0.668. The visualized comparison between the size of the dataset can be seen in Figure 4 for the validation data and in Figure 6 for the testing data. Despite this, there is no significant performance increase, suggesting that even with the smallest dataset, satisfactory performance in training, but not in testing is achieved. However, it is possible that more advanced models, such as newer versions of YOLO, could achieve better test performance.

Figure 4: mAP@0.5 and mAP@0.5:0.95 metrics of validation set of all five dataset

Figure 5: mAP@0.5 and mAP@0.5:0.95 metrics of Testing set of all five dataset