Li-ViP3D: Enhancing End-to-End Perception and Prediction with Camera-LiDAR Fusion

Solving the problem of perception and subsequent trajectory prediction of various agents in a scene is crucial for self-driving vehicles and other autonomous systems. Traditionally, this process has been simplified by splitting perception and prediction into separate steps, leading to limited information flow and error propagation between modules. While such modular approaches have their benefits, many modern approaches instead focus on unified, end-to-end, perception and prediction (PnP) aiming to prevent the propagation of errors and decisions made based on limited context. In this work, we focus on addressing one of the main limitations of current PnP solutions, the insufficient exploitation of the complementarity between camera and LiDAR data. While modern approaches extract a vast amount of information from standard camera data, they often overlook the benefits of integrating it with LiDAR data for improved robustness and accuracy. In this work, we extend ViP3D, a query-based vision model for trajectory prediction, by incorporating LiDAR data to enhance scene understanding and prediction quality. Our approach, LiViP3D, fuses visual and LiDAR modalities in an end-to-end differentiable framework, leveraging their combined strengths to detect, track, and predict agent trajectories with greater precision. ... mehrWe validate our approach on the large-scale nuScenes dataset, demonstrate the viability of query-based approaches for camera-lidar multimodal PnP, and show improvements over the base ViP3D model and other end-to-end approaches.