3D Voxel Reconstruction and World Model for Autonomous Driving

The real world is complex, variable, and highly interactive. Therefore, for autonomous driving, understanding the scene and predicting plausible futures is crucial. World models have the capability to learn the environment through observations and make predictions based on actions. Also, the choice of sensors is very important in order to get comprehensive information about the environment. However, existing world models mostly use low resolution images as input and operate only in simple environments.

To address this challenge, we propose an innovative world model that combines point cloud data from the LiDAR and RGB image data from the camera sensor as observation inputs. Additionally, the model not only decodes observations from latent states but also reconstructs 3D voxel occupancy as outputs to enhance the understanding of the environment's geometric features. Unlike most existing world models, our model not only contains 2D visual information but also incorporates 3D spatial data and represents the environment more accurately and comprehensively through voxels.

In our study, we implemented various methods to extract features from point clouds and images, analyzing their effects. ... mehrWe used a transformer-based architecture to fuse features from different sensors and compared it to two other simple methods: one is directly averaging features from various sensors, while the other is concatenating features followed by a fully connected layer. Moreover, we adapted the model to 2D latent states. We collected training and evaluation data in the CARLA simulation environment. Then, we evaluated the reconstruction and prediction performance of different combinations of feature extraction and feature fusion methods, as well as 1D and 2D latent states. The study demonstrates the good performance of our novel world model framework in reconstruction and prediction.