AI in multispectral image analysis: Implementing a deep learning model for the segmentation of common thermal urban features to assist in the automation of infrastructure-related maintenance

Thermography has proven to be a powerful tool in infrastructure-related maintenance, such as building inspections, industrial applications, and the detection of anomalous occurrences like fires, gas, or heating pipeline leakages. Meanwhile, technological advancements in unmanned aerial flight have paved the way for automating multi-spectral image acquisition for these applications.

As a powerful and versatile data handling tool, AI offers the means for the consequent analysis of such RGB-T data. Both machine learning (ML) and deep learning (DL) models have been shown to perform well in classifying different kinds of aerial spectral images to various ends, such as city information modeling, thermal bridge detection or tree classification. However, these studies mostly use satellite or vehicle-based data acquired at daytime, which – owing to high overall temperatures – reduces the usefulness of the information provided by the thermal channel.

This presentation describes the implementation of an AI model in an attempt to segment and thus classify various common urban and thermal features in a novel RGB-T dataset. The dataset, comprising 793 images from two German cities, was acquired by automatic unmanned aerial vehicle (UAV) flight at nighttime and various features common to the urban context – such as buildings, cars, manholes, etc. ... mehr– were manually annotated.

On account of nighttime acquisition, the images are preprocessed to increase contrast and illumination as well as remove distortive effects in the thermal channel. A U-Net architecture is selected to perform the segmentation task due to its ability to handle small datasets and prevalence in literature. Various model backbone and hyperparameter tuning experiments help identify the best performing model, which is evaluated using a broad range of metrics relevant in semantic image segmentation, including accuracy, f1 score, and mean intersection over union.

This classification of various features in the urban context can be used to separate common thermal occurrences from relevant anomalies and thus contribute to the automation of maintenance-related processes in urban infrastructure. Expanding the dataset with more annotations, especially of underrepresented classes, will help improve the existing model.