Speedup of Hyperparameter Optimization in Propulate Using Approximative Surrogate Models

Hyperparameter optimization (HPO) is a critical aspect in machine learning (ML) that involves finding the most effective settings for a model’s non-trainable parameters, which can significantly impact its performance. Automated approaches like Random Search, Grid Search, Bayesian Optimization (BO), or evolutionary algorithms (EAs) train the neural network (NN) over and over again, testing new hyperparameters (HPs) every time. This is exceptionally compute-intensive, especially as newer models get bigger and bigger. Predicting the performance of HP configurations during the training process allows for early termination of less promising configurations. This work introduces surrogate models (SMs) into Propulate, a program designed for HPO in high performance computing (HPC) environments. SM have access to interim loss values from each evaluated NN’s training during the HPO and decide about stopping it early. Evaluating static and probabilistic SMs for HPO in Propulate with different datasets and NNs shows a significant decrease in total run time and energy consumption while still finding a loss within small bounds of the best loss found without early stopping. ... mehrThe new SM implementation added to Propulate is universally usable. It enables finding and developing more powerful SMs in the future.