An empirical comparison of V -fold penalisation and cross-validation for model selection in distribution-free regression

Abstract : Model selection is a crucial issue in machine-learning and a wide variety of penalisation methods (with possibly data dependent complexity penalties) have recently been introduced for this purpose. However their empirical performance is generally not well documented in the literature. It is the goal of this paper to investigate to which extent such recent techniques can be successfully used for the tuning of both the regularisation and kernel parameters in support vector regression (SVR) and the complexity measure in regression trees (CART). This task is traditionally solved via V-fold cross-validation (VFCV), which gives efficient results for a reasonable computational cost. A disadvantage however of VFCV is that the procedure is known to provide an asymptotically suboptimal risk estimate as the number of examples tends to infinity. Recently, a penalisation procedure called V-fold pe-nalisation has been proposed to improve on VFCV, supported by theoretical arguments. Here we report on an extensive set of experiments comparing V-fold penalisation and VFCV for SVR/CART calibration on several benchmark datasets. We highlight cases in which VFCV and V-fold penalisation provide poor estimates of the risk respectively and introduce a modified penalisation technique to reduce the estimation error.
Type de document :
Article dans une revue
Pattern Analysis and Applications, Springer Verlag, 2016, 19 (1), pp.41-53. 〈10.1007/s10044-014-0381-y〉
Liste complète des métadonnées

Littérature citée [36 références]  Voir  Masquer  Télécharger

https://hal.sorbonne-universite.fr/hal-01567869
Contributeur : Nicolas Baskiotis <>
Soumis le : lundi 24 juillet 2017 - 16:02:27
Dernière modification le : mercredi 21 mars 2018 - 18:58:13

Fichier

1212.1780.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Charanpal Dhanjal, Nicolas Baskiotis, Stéphan Clémençon, Nicolas Usunier. An empirical comparison of V -fold penalisation and cross-validation for model selection in distribution-free regression. Pattern Analysis and Applications, Springer Verlag, 2016, 19 (1), pp.41-53. 〈10.1007/s10044-014-0381-y〉. 〈hal-01567869〉

Partager

Métriques

Consultations de la notice

126

Téléchargements de fichiers

34