PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers - TETIS Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers

Résumé

Computer vision methods that explicitly detect object parts and reason on them are a step towards inherently interpretable models. Existing approaches that perform part discovery driven by a fine-grained classification task make very restrictive assumptions on the geometric properties of the discovered parts; they should be small and compact. Although this prior is useful in some cases, in this paper we show that pre-trained transformer-based vision models, such as self-supervised DINOv2 ViT, enable the relaxation of these constraints. In particular, we find that a total variation (TV) prior, which allows for multiple connected components of any size, substantially outperforms previous work. We test our approach on three fine-grained classification benchmarks: CUB, PartImageNet and Oxford Flowers, and compare our results to previously published methods as well as a re-implementation of the state-of-the-art method PDiscoNet with a transformer-based backbone. We consistently obtain substantial improvements across the board, both on part discovery metrics and the downstream classification task, showing that the strong inductive biases in self-supervised ViT models require to rethink the geometric priors that can be used for unsupervised part discovery.
Fichier principal
Vignette du fichier
11397.pdf (7.36 Mo) Télécharger le fichier
11397-supp.pdf (4.39 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
licence
licence

Dates et versions

hal-04659631 , version 1 (23-07-2024)

Licence

Identifiants

Citer

Ananthu Aniraj, Cassio F Dantas, Dino Ienco, Diego Marcos. PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers. ECCV 2024 - 18th European Conference on Computer Vision, Sep 2024, Milano, Italy. ⟨hal-04659631⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More