Communication Dans Un Congrès Année : 2025

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

Résumé

This study investigates the behavior of model-integrated routers in Mixture of Experts (MoE) models, focusing on how tokens are routed based on their linguistic features, specifically Part-of-Speech (POS) tags. The goal is to explore across different MoE architectures whether experts specialize in processing tokens with similar linguistic traits. By analyzing token trajectories across experts and layers, we aim to uncover how MoE models handle linguistic information. Findings from six popular MoE models reveal expert specialization for specific POS categories, with routing paths showing high predictive accuracy for POS, highlighting the value of routing paths in characterizing tokens.
Fichier principal
Vignette du fichier
2025.coling-main.431.pdf (3) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04918227 , version 1 (29-01-2025)

Identifiants

Citer

Elie Antoine, Frédéric Béchet, Philippe Langlais. Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models. Proceedings of the 31st International Conference on Computational Linguistics, Association for Computational Linguistics, Jan 2025, Abu Dhabi, United Arab Emirates. pp.6467-6474. ⟨hal-04918227⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

More