Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning - CNRS - Centre national de la recherche scientifique Accéder directement au contenu
Article Dans Une Revue Neural Computation Année : 2020

Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning

Résumé

Sensory processing is increasingly conceived in a predictive framework in which neurons would constantly process the error signal resulting from the comparison of expected and observed stimuli. Surprisingly, few data exist on the accuracy of predictions that can be computed in real sensory scenes. Here, we focus on the sensory processing of auditory and audiovisual speech. We propose a set of computational models based on artificial neural networks (mixing deep feedforward and convolutional networks), which are trained to predict future audio observations from present and past audio or audiovisual observations (i.e., including lip movements). Those predictions exploit purely local phonetic regularities with no explicit call to higher linguistic levels. Experiments are conducted on the multispeaker LibriSpeech audio speech database (around 100 hours) and on the NTCD-TIMIT audiovisual speech database (around 7 hours). They appear to be efficient in a short temporal range (25–50 ms), predicting 50% to 75% of the variance of the incoming stimulus, which could result in potentially saving up to three-quarters of the processing power. Then they quickly decrease and almost vanish after 250 ms. Adding information on the lips slightly improves predictions, with a 5% to 10% increase in explained variance. Interestingly the visual gain vanishes more slowly, and the gain is maximum for a delay of 75 ms between image and predicted sound.
Fichier principal
Vignette du fichier
Hueber.pdf (846.34 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03016083 , version 1 (25-11-2020)

Identifiants

Citer

Thomas Hueber, Eric Tatulli, Laurent Girin, Jean-Luc Schwartz. Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning. Neural Computation, 2020, 32 (3), pp.596-625. ⟨10.1162/neco_a_01264⟩. ⟨hal-03016083⟩
120 Consultations
183 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More