RF+clust for Leave-One-Problem-Out Performance Prediction - Sorbonne Université Access content directly
Conference Papers Year : 2023

RF+clust for Leave-One-Problem-Out Performance Prediction

Ana Nikolikj
Carola Doerr
Tome Eftimov

Abstract

Per-instance automated algorithm configuration and selection are gaining significant moments in evolutionary computation in recent years. Two crucial, sometimes implicit, ingredients for these automated machine learning (AutoML) methods are 1) feature-based representations of the problem instances and 2) performance prediction methods that take the features as input to estimate how well a specific algorithm instance will perform on a given problem instance. Nonsurprisingly, common machine learning models fail to make predictions for instances whose feature-based representation is underrepresented or not covered in the training data, resulting in poor generalization ability of the models for problems not seen during training. In this work, we study leave-one-problem-out (LOPO) performance prediction. We analyze whether standard random forest (RF) model predictions can be improved by calibrating it with a weighted average of performance values obtained by the algorithm on problem instances that are sufficiently close to the problem for which a performance prediction is sought, measured by cosine similarity in feature space. While our RF+clust approach obtains more accurate performance prediction for several problems, its predictive power crucially depends on the chosen similarity threshold as well as on the feature portfolio for which the cosine similarity is measured, thereby opening a new angle for feature selection in a zero-shot learning setting, as LOPO is termed in machine learning.
Fichier principal
Vignette du fichier
RFclust_EvoApps_HAL.pdf (3.77 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04180591 , version 1 (12-08-2023)

Identifiers

Cite

Ana Nikolikj, Carola Doerr, Tome Eftimov. RF+clust for Leave-One-Problem-Out Performance Prediction. Applications of Evolutionary Computation (EvoApplications 2023), Apr 2023, Brno, Czech Republic. pp.285-301, ⟨10.1007/978-3-031-30229-9_19⟩. ⟨hal-04180591⟩
18 View
9 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More