Novel View Synthesis Under Rainy Conditions with Neural Radiance Fields and Gaussian Splatting

Scene reconstruction and novel view synthesis from calibrated multi-view images still attracts a lot of attention in computer vision and graphics. However, the assumption that images are noise-free rarely holds in real-world scenarios where adverse weather conditions are inevitable. Being a part of our environment, we are particularly interested in rain as dynamic semi-transparent occlusion which imposes challenges to a complete and accurate geometry of the underlying features. More precisely, we qualitatively and quantitatively analyze the photometric image quality under rainy conditions generated by radiance field methods, namely: Neural Radiance Fields (NeRFs), 3D Gaussian Splatting (3DGS) and 2D Gaussian Splatting (2DGS) due to the different geometric representation. To assess the impact of rain to the scene reconstruction we consider raindrops and streaks captured with illumination variation as well as occlusion masks with different coverage. The evaluation is based on comparing 2D image metrics of the rendered novel views without and with masks. The experiments and results show that 3DGS achieves highest rendering fidelity in all scenarios without and with masks with SSIM of 0.724 and LPIPS of 0.291, followed by 2DGS with slightly lower scores, while NeRF exhibits lowest correspondence with the input images with SSIM of 0.584 and LPIPS of 0.384. ... mehrWe demonstrate the effectiveness of using masks to handle rain as transient element and radiance field methods’ ability to reliably approximate the geometry behind rain occlusions.