Rethinking Hierarchical Supervision: Revisiting Simplicity in the Era of Strong Visual Backbones

Fine-grained visual classification is difficult due to high inter-class similarity and intra-class variation requiring large amounts of data. Yet many datasets (e.g., in biodiversity) provide a taxonomic hierarchy (genus/family/order) as additional training label for free which is rarely used. We revisit lightweight hierarchy exploitation under modern visual backbones and show that simple methods can be effective again. Our training recipe called CHiMT combines two complementary objectives with a shared backbone: (i) hierarchy-supportive multi-task supervision across taxonomy levels and (ii) hierarchy-aware contrastive learning with taxonomy-driven negative sampling that prioritizes nearby taxa. On multiple iNaturalist2021 subsets, these components consistently improve fine-grained Top-1 accuracy and reduce hierarchical mistake severity (LCA). Contrastive learning also strengthens open-set performance (AUROC). The combined approach yields the most consistent gains without inference-time overhead. We further show that soft pooling often improves coarse-level predictions and that hierarchy-aware training partially mitigates coarse-level degradation in an open-set coarse-known setting. ... mehrCode: https://github.com/ThelPhi/chimt_fgic.