TMF: Temporal multi-modal fusion framework for estimating wheat yield from multi-source satellite and environmental data across the European Union

Accurate large-scale crop yield estimation is critical for agricultural management and advance warning of potentially compromised food security. While recent advances in remote sensing provide diverse data sources for crop monitoring, effectively integrating these data sources for accurate yield estimation at the continental scale remains challenging due to complex crop-environment interactions and high spatial heterogeneity. In this study, we developed a Temporal Multi-modal Fusion (TMF) framework for end-of-season wheat yield estimation at the sub-national level across the European Union from 2001 to 2019. Our framework integrated time-series data across the entire growing season from climate variables, satellite-based remote sensing measurements including vegetation indices (e.g., Enhanced Vegetation Index) and productivity indicators (e.g., Solar-Induced Fluorescence, Gross Primary Productivity), and static soil properties. By employing parallel transformer encoders followed by a late-fusion strategy, our architecture preserves modality-specific temporal dynamics before explicitly anchoring them against static spatial constraints. ... mehrLeave-one-year-out cross-validation demonstrated that the TMF framework achieved accurate yield estimation with an RMSE of 0.75 Mg·ha$^{−1}$, 7–38% lower than baseline models (LSTM, GBRT, RF, and ANN). The model demonstrated better spatial stability and captured severe yield anomalies during extreme climate events (e.g., the 2003 and 2018 droughts). The ablation study and SHAP-based interpretability analysis revealed that while fusing more input modalities consistently improved model performance, substantial information redundancy exists among multi-modal inputs. Although satellite-derived measurements effectively capture seasonal canopy dynamics and interannual yield fluctuations, static soil properties provide complementary spatially structured information for yield estimation. Their integration reduced RMSE by 28–33% compared to climate-only inputs and improved prediction stability across the study domain. Furthermore, the model’s temporal contribution patterns aligned with wheat phenology, with feature importance peaking during reproductive and grain-filling stages. These findings highlight that the TMF framework provides a robust and scalable approach for continental-scale crop monitoring across highly diverse agroecosystems.