Contextual Word Embeddings Clustering Through Multiway Analysis: A Comparative Study - CNRS - Centre national de la recherche scientifique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Contextual Word Embeddings Clustering Through Multiway Analysis: A Comparative Study

Mira Ait-Saada
  • Fonction : Auteur
  • PersonId : 1221323
Mohamed Nadif
  • Fonction : Auteur
  • PersonId : 1134506

Résumé

Transformer-based contextual word embedding models are widely used to improve several NLP tasks such as text classification and question answering. Knowledge about these multi-layered models is growing in the literature, with several studies trying to understand what is learned by each of the layers. However, little is known about how to combine the information provided by these different layers in order to make the most of the deep Transformer models. On the other hand, even less is known about how to best use these modes for unsupervised text mining tasks such as clustering. We address both questions in this paper, and propose to study several multiway-based methods for simultaneously leveraging the word representations provided by all the layers. We show that some of them are capable to perform word clustering in an effective and interpretable way. We evaluate their performances across a wide variety of Transformer models, datasets, multiblock techniques and tensor-decomposition methods commonly used to tackle three-way data.
Fichier principal
Vignette du fichier
IDA_2023_ait-saada.pdf (569.35 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04492955 , version 1 (08-04-2024)

Identifiants

Citer

Mira Ait-Saada, Mohamed Nadif. Contextual Word Embeddings Clustering Through Multiway Analysis: A Comparative Study. 21st International Symposium on Intelligent Data Analysis (IDA 2023), Apr 2023, Louvain-la-Neuve, France. pp.1-14, ⟨10.1007/978-3-031-30047-9_1⟩. ⟨hal-04492955⟩
19 Consultations
8 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More