Article Dans Une Revue Transactions on Machine Learning Research Journal Année : 2025

Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization

Résumé

In recent years, bilevel approaches have become very popular to efficiently estimate high-dimensional hyperparameters of machine learning models. However, to date, binary parameters are handled by continuous relaxation and rounding strategies, which could lead to inconsistent solutions. In this context, we tackle the challenging optimization of mixed-binary hyperparameters by resorting to an equivalent continuous bilevel reformulation based on an appropriate penalty term. We propose an algorithmic framework that, under suitable assumptions, is guaranteed to provide mixed-binary solutions. Moreover, the generality of the method allows to safely use existing continuous bilevel solvers within the proposed framework. We evaluate the performance of our approach for two specific machine learning problems, i.e., the estimation of the group-sparsity structure in regression problems and the data distillation problem. The reported results show that our method is competitive with state-of-the-art approaches based on relaxation and rounding

Mots clés

Fichier principal
Vignette du fichier
2025_Venturini_S_j-tmlr_mib.pdf (787) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04183917 , version 1 (21-08-2023)
hal-04183917 , version 2 (16-01-2025)
hal-04183917 , version 3 (10-02-2025)

Licence

Identifiants

Citer

Sara Venturini, Marianna de Santis, Jordan Patracone, Francesco Rinaldi, Saverio Salzo, et al.. Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization. Transactions on Machine Learning Research Journal, 2025. ⟨hal-04183917v3⟩
31 Consultations
74 Téléchargements

Altmetric

Partager

More