Embedding Arithmetic of Multimodal Queries for Image Retrieval - CNRS - Centre national de la recherche scientifique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Embedding Arithmetic of Multimodal Queries for Image Retrieval

Résumé

Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing pairwise relationships between scene elements. The goal is to retrieve an image consistent with the (source image, text transformation) query. We use an image/text matching oracle (OSCAR) to assess whether the image transformation is successful. The SIMAT dataset will be publicly available. We use SIMAT to evaluate the geometric properties of multimodal embedding spaces trained with an image/text matching objective, like CLIP. We show that vanilla CLIP embeddings are not very well suited to transform images with delta vectors, but that a simple finetuning on the COCO dataset can bring dramatic improvements. We also study whether it is beneficial to leverage pretrained universal sentence encoders (FastText, LASER and LaBSE).
Fichier principal
Vignette du fichier
embedding_arithmetic_arxiv_2023.pdf (1.53 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03957468 , version 1 (26-01-2023)

Licence

Paternité - Pas d'utilisation commerciale

Identifiants

Citer

Guillaume Couairon, Matthijs Douze, Matthieu Cord, Holger Schwenk. Embedding Arithmetic of Multimodal Queries for Image Retrieval. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun 2022, New Orleans, France. pp.4946-4954, ⟨10.1109/CVPRW56347.2022.00542⟩. ⟨hal-03957468⟩
3 Consultations
57 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More