Improving the Generalization of Supervised Models - Laboratoire Jean Kuntzmann Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

Improving the Generalization of Supervised Models

Résumé

We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at that task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization while maintaining its performance on the original task. Models trained with self-supervised learning (SSL) tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We enrich the common supervised training framework using two key components of recent SSL models: multi-scale crops for data augmentation and the use of an expendable projector head. We replace the last layer of class weights with class prototypes computed on the fly using a memory bank. We show that these three improvements lead to a more favorable trade-off between the IN1K training task and 13 transfer tasks. Over all the explored configurations, we single out two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks. Project page and pretrained models: https://europe.naverlabs.com/t-rex.
Fichier principal
Vignette du fichier
t_ReX.pdf (1.53 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03929621 , version 1 (08-01-2023)
hal-03929621 , version 2 (10-03-2023)

Identifiants

Citer

Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, Diane Larlus. Improving the Generalization of Supervised Models. 2022. ⟨hal-03929621v1⟩
292 Consultations
188 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More