More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification - CNRS - Centre national de la recherche scientifique Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Geoscience and Remote Sensing Année : 2021

More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification

Danfeng Hong
Lianru Gao
Naoto Yokoya
Jing Yao
Qian Du
Bing Zhang

Résumé

Classification and identification of the materials lying over or beneath the earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS), and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML)-cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on “what,” “where,” and “how” to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS data sets. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

Dates et versions

hal-03429642 , version 1 (15-11-2021)

Identifiants

Citer

Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, et al.. More Diverse Means Better: Multimodal Deep Learning Meets Remote-Sensing Imagery Classification. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (5), pp.4340-4354. ⟨10.1109/TGRS.2020.3016820⟩. ⟨hal-03429642⟩
68 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More