DICE: DIVERSITY IN DEEP ENSEMBLES VIA CONDITIONAL REDUNDANCY ADVERSARIAL ESTIMATION - CNRS - Centre national de la recherche scientifique Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

DICE: DIVERSITY IN DEEP ENSEMBLES VIA CONDITIONAL REDUNDANCY ADVERSARIAL ESTIMATION

Alexandre Ramé
  • Fonction : Auteur
  • PersonId : 1120615
  • IdRef : 276144511
Matthieu Cord

Résumé

Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members' performances. In this paper, we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies. Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional mutual information, we introduce a novel training criterion called DICE: it increases diversity by reducing spurious correlations among features. The main idea is that features extracted from pairs of members should only share information useful for target class prediction without being conditionally redundant. Therefore, besides the classification loss with information bottleneck, we adversarially prevent features from being conditionally predictable from each other. We manage to reduce simultaneous errors while protecting class information. We obtain state-of-the-art accuracy results on CIFAR-10/100: for example, an ensemble of 5 networks trained with DICE matches an ensemble of 7 networks trained independently. We further analyze the consequences on calibration, uncertainty estimation, out-of-distribution detection and online co-distillation.
Fichier principal
Vignette du fichier
dice_diversity_in_deep_ensembl.pdf (6.89 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03483364 , version 1 (16-12-2021)

Identifiants

  • HAL Id : hal-03483364 , version 1

Citer

Alexandre Ramé, Matthieu Cord. DICE: DIVERSITY IN DEEP ENSEMBLES VIA CONDITIONAL REDUNDANCY ADVERSARIAL ESTIMATION. ICLR 2021 - 9th International Conference on Learning Representations, May 2021, Vienne, Austria. ⟨hal-03483364⟩
24 Consultations
29 Téléchargements

Partager

Gmail Facebook X LinkedIn More