KIT | KIT-Bibliothek | Impressum | Datenschutz

Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks

Kattenborn, Teja; Schiefer, Felix ORCID iD icon 1; Frey, Julian; Feilhauer, Hannes; Mahecha, Miguel D.; Dormann, Carsten F.
1 Institut für Geographie und Geoökologie (IFGG), Karlsruher Institut für Technologie (KIT)

Abstract (englisch):

Deep learning and particularly Convolutional Neural Networks (CNN) in concert with remote sensing are becoming standard analytical tools in the geosciences. A series of studies has presented the seemingly outstanding performance of CNN for predictive modelling. However, the predictive performance of such models is commonly estimated using random cross-validation, which does not account for spatial autocorrelation between training and validation data. Independent of the analytical method, such spatial dependence will inevitably inflate the estimated model performance. This problem is ignored in most CNN-related studies and suggests a flaw in their validation procedure. Here, we demonstrate how neglecting spatial autocorrelation during cross-validation leads to an optimistic model performance assessment, using the example of a tree species segmentation problem in multiple, spatially distributed drone image acquisitions. We evaluated CNN-based predictions with test data sampled from 1) randomly sampled hold-outs and 2) spatially blocked hold-outs. Assuming that a block cross-validation provides a realistic model performance, a validation with randomly sampled holdouts overestimated the model performance by up to 28%. ... mehr


Verlagsausgabe §
DOI: 10.5445/IR/1000156879
Veröffentlicht am 13.03.2023
Originalveröffentlichung
DOI: 10.1016/j.ophoto.2022.100018
Scopus
Zitationen: 38
Dimensions
Zitationen: 49
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Geographie und Geoökologie (IFGG)
Publikationstyp Zeitschriftenaufsatz
Publikationsmonat/-jahr 08.2022
Sprache Englisch
Identifikator ISSN: 2667-3932
KITopen-ID: 1000156879
Erschienen in ISPRS Open Journal of Photogrammetry and Remote Sensing
Verlag Elsevier
Band 5
Seiten Art.-Nr.: 100018
Vorab online veröffentlicht am 21.06.2022
Schlagwörter Spatial autocorrelation; Convolutional neural networks; Deep learning; Machine learning; Mapping; Reference data
Nachgewiesen in Dimensions
Scopus
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page