Anomaly Detection in 3D Space for Autonomous Driving

Current state-of-the-art perception models do not always detect all objects in an image. Therefore, they cannot currently be relied upon in safety critical applications such as autonomous driving. Objects that cannot be detected are called anomalies. Current work on anomaly detection is primarily based on camera data. This work evaluates to what extent it is possible today to do anomaly detection in 3D on pseudo-lidar data. A pseudo-lidar is a model that estimates 3D depth for each pixel of an image. Currently, there is no approach available that performs anomaly detection on pseudo-lidar data.

Research Question 1 (RQ1) considers whether dissimilarities between lidar and pseudo-lidar are an indicator of anomalies. For this purpose, it is evaluated whether there are larger deviations between pseudo-lidar and lidar point clouds for anomalies compared to non-anomalies. There is no multi-modal dataset for anomaly detection available which could be directly used. Therefore, in the multi-modal KITTI-360 dataset each instance that was not correctly segmented by a panoptic segmentation model, was labeled as anomaly. It is shown how the anomaly definition depends on the used segmentation criterion. ... mehrThe dataset contains 2D images with instance labels and corresponding 3D lidar point clouds with corresponding instance labels. With these labels, one can compare pseudo-lidar and lidar instance by instance. The 2D instance labels allow to define which instances are correctly segmented. In a next step the chamfer-distance between point clouds of each instance in lidar and pseudo-lidar are calculated. Lidar was used as physically captured ground truth. It has been found that the deviation between lidar and pseudo-lidar is similar for anomalies and non-anomalies. Thus, the dissimilarity between lidar and pseudo-lidar can not be used as an indicator for an anomaly.

In Research Question 2 (RQ2) it was analyzed how good a pseudo-lidar can map anomalies of type novelties in 3D. Therefore, one augmented anomaly dataset, and two real world anomaly datasets were considered. All these datasets are image based. For answering the research questions, both a quantitative and qualitative analysis was carried out. As a quantitative analysis, Monte Carlo Dropout was applied onto these datasets to evaluate the uncertainty of the model. The 3D point clouds estimated with the pseudo-lidar were visualized for the qualitative analysis. The qualitative analysis shows that some anomalies can be mapped well in 3D and others are not mapped at all. Furthermore, it is shown that augmented anomalies can be sometimes mapped ambiguously in 3D. In the quantitative analysis, it is shown for all datasets considered that the pseudo-lidar is more certain for anomaly regions than for non-anomaly regions which is interpreted as a consequence of an overconfidence of the model. Furthermore, the anomaly concept is not always consistent across different modalities.

The Research Question 3 (RQ3) analyzed whether anomalies can be found using flow estimation on pseudo-lidar predicted point clouds. An anomaly would be present in theory if the motion segmentation model contradicted with a panoptic segmentation model. For this purpose, it was investigated whether the pseudo-lidar estimated point clouds are consistent enough through time to do flow estimation on them. For this purpose, the multi-modal KITTI-360 dataset was used.
For each instance in the pseudo-lidar it was determined how much a point cloud of an instance differs from the same instance in the next frame. For the consistency evaluation of static and dynamic instances, the ego motion has to be extracted. The pseudo-lidar prediction is consistent between frames if for static instances the distance is small and for dynamic instances the distance is equal to the motion of the instance. It has been shown that the pseudo-lidar makes inconsistent predictions over time, and therefore one cannot distinguish between static and dynamic instances based on pseudo-lidar point clouds. It follows that a flow-based approach to anomaly detection is not possible for point clouds predicted by current single image based pseudo-lidars.