KIT | KIT-Bibliothek | Impressum | Datenschutz

Why Do Machine Learning Practitioners Still Use Manual Tuning? A Qualitative Study

Hasebrook, Niklas; Morsbach, Felix 1; Kannengießer, Niclas ORCID iD icon 1; Zöller, Marc; Franke, Jörg; Lindauer, Marius; Hutter, Frank; Sunyaev, Ali 1
1 Institut für Angewandte Informatik und Formale Beschreibungsverfahren (AIFB), Karlsruher Institut für Technologie (KIT)

Abstract:

Advanced programmatic hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sample efficiency in reproducibly finding optimal hyperparameter values of machine learning (ML) models. Yet, ML practitioners often apply less sample-efficient HPO methods, such as grid search, which often results in under-optimized ML models. As a reason for this behavior, we suspect practitioners choose HPO methods based on individual motives, consisting of contextual factors and individual goals. However, practitioners' motives still need to be clarified, hindering the evaluation of HPO methods for achieving specific goals and the user-centered development of HPO tools. To understand practitioners' motives for using specific HPO methods, we used a mixed-methods approach involving 20 semi-structured interviews and a survey study with 71 ML experts to gather evidence of the external validity of the interview results. By presenting six main goals (e.g., improving model understanding) and 14 contextual factors affecting practitioners' selection of HPO methods (e.g., available computer resources), our study explains why practitioners use HPO methods that seem inappropriate at first glance. ... mehr


Volltext (Version 2) §
DOI: 10.5445/IR/1000143445/v2
Veröffentlicht am 22.08.2023
Volltext (Version 1) §
DOI: 10.5445/IR/1000143445
Veröffentlicht am 07.03.2022
Cover der Publikation
Zugehörige Institution(en) am KIT Institut für Angewandte Informatik und Formale Beschreibungsverfahren (AIFB)
Kompetenzzentrum für angewandte Sicherheitstechnologie (KASTEL)
Publikationstyp Forschungsbericht/Preprint
Publikationsdatum 02.03.2022
Sprache Englisch
Identifikator KITopen-ID: 1000143445
HGF-Programm 46.23.01 (POF IV, LK 01) Methods for Engineering Secure Systems
Verlag Karlsruher Institut für Technologie (KIT)
Schlagwörter Machine Learning, Hyperparameter Optimization, Automated Machine Learning
Nachgewiesen in Dimensions
arXiv
KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft
KITopen Landing Page