How Does Feature Engineering Impact UAV-based Multispectral Semantic Segmentation? An RGB and Thermal Image Ablation Study

While deep learning (DL) has become indispensable for standard computer vision tasks, the growing availability of sophisticated sensors has fuelled interest in applications using more unconventional data, such as thermal infrared (TIR) imagery. Thermography is a valuable tool for numerous heat-related applications, although the resulting images are particularly susceptible to noise, artefacts, and low resolution. Publications applying DL to multispectral data focus on enhancing existing models to tackle quality issues without considering the imagery itself.
This work therefore explores how manual feature engineering can influence model performance when transferring prevalent architectures to combined red green blue (RGB) and TIR imagery. Utilising the popular U-Net model and a novel unmanned aerial vehicle (UAV)-based dataset from two German cities, we address the common remote sensing task of multi-class semantic segmentation – specifically thermal urban feature detection to support energy-related maintenance. A comprehensive ablation study of various filters, channel constellations, and image sizes helps investigate how loss of information about i.e. ... mehrcolour or resolution information impacts model performance. Various common semantic segmentation metrics are compared to find the most suitable feature engineering. Measuring resource usage – especially energy consumption – enables an assessment in terms of sustainability.
The choice of ablation study parameters are found to have a 7% to 10% impact on performance metrics. Resource utilisation depends on image size with the relationship following a logarithmic growth curve. While added RGB information improves performance when working with raw imagery, the overall best results are achieved using only TIRs with in-depth feature engineering. A higher resolution is found to yield the best results most often and improves those metrics that perform worst on the imbalanced dataset. This is found to require surprising little additional resources.