摘要详情

ID / 提交时间

52 / 2023-08-30 12:28:25

标题

Multimodal sentiment analysis missing modality reconstruction network based on shared-specific features

关键字

multimodal sentiment analysis,missing modal,coherence features

主题及专题

Special Sessions > SS02 - Intelligent Robot, Artificial Intelligence Methods and Its Applications

状态

终稿

作者

Yu Guo / Dongguan University of Technology

Yong Qin / Dongguan University of Technology

Xuwen Qin / Shenzhen International Exchange College

Ziliang Ren / Dongguan University of Technology

Lei Chen / Dongguan University of Technology

摘要

In multimodal sentiment analysis, heterogeneity between modalities makes inconsistent modal distributions a challenge. Especially in the case of incomplete features of certain modalities, the differences between modalities may interfere with the accurate prediction of sentiment categories. To address this problem, this paper proposes a missing modal reconstruction network (SSF-MMRN) for multimodal sentiment analysis based on sharing specific features.Firstly, a CMD distance-constrained training strategy is used to learn inter-modal consistency features. Second, based on the consistency features, a reconstruction module is proposed to generate missing modal features, check the semantic consistency of the recovered modalities with the original available modalities, and introduce inconsistencies into multiple models for better decision-making once they exist. Extensive experiments on the IEMOCAP benchmark dataset show that our proposed model effectively mitigates the modality gap during missing modality prediction and significantly improves the emotion recognition performance.