A second-order-like optimizer with adaptive gradient scaling for deep learning - Argumentation, Décision, Raisonnement, Incertitude et Apprentissage
Preprints, Working Papers, ... Year : 2024

A second-order-like optimizer with adaptive gradient scaling for deep learning

Abstract

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.
Fichier principal
Vignette du fichier
innaprop/innaprop_arxiv.pdf (971.87 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04724894 , version 1 (07-10-2024)

Identifiers

Cite

Jérôme Bolte, Ryan Boustany, Edouard Pauwels, Andrei Purica. A second-order-like optimizer with adaptive gradient scaling for deep learning. 2024. ⟨hal-04724894⟩
94 View
76 Download

Altmetric

Share

More