Multi-objective selection of algorithm portfolios

We propose a method for selecting a portfolio of algorithms optimizing multiple criteria. We select a portfolio of limited size and at the same time good quality from a possibly large pool of algorithms. Our method also helps to decide which algorithm to use for each trade-off between conflicting objectives. Many algorithms depend on a number of parameters and, therefore, require problem-specific tuning for a suitable performance. In multi-objective tuning, different parameter settings of one algorithm will lead to different trade-offs between the conflicting objectives. Hence, discrete approximations of the corresponding Pareto front resulting from different parameter settings of each algorithm must be compared. Our technique is applied post-hoc to these approximations. It discards algorithms that contribute only insignificantly to the overall Pareto front and delivers simple and interpretable decision rules which algorithm to choose based on the desired trade-off. The new method is applied to the selection of approximative support vector machine solvers, where the objectives are high accuracy and short training time. The analysis hints at dropping several solvers completely and yields insights into the specific strengths of the remaining solvers.