Multimodal Customer Emotion Recognition in Social Media Support: Integrating Textual Sentiment with Interaction Metadata via Graph Neural Networks

• Abey Litty

  Texas University

  Author

capability for organizations seeking to understand and respond to customer needs effectively. Traditional approaches predominantly rely on unimodal textual sentiment analysis, overlooking the rich relational context embedded in customer-agent interaction patterns and metadata. This study addresses this gap by proposing a multimodal framework that integrates textual sentiment analysis with interaction metadata through Graph Neural Networks (GNNs). The methodology employs BERT-based embeddings for textual feature extraction, constructs a heterogeneous graph structure capturing user-customer-agent interactions, temporal patterns, and response histories, and applies Graph Attention Networks (GAT) for relational sentiment propagation. Evaluation on a dataset of over 6,500 social media customer service conversations demonstrates that the proposed framework achieves an accuracy of 89.4%, outperforming traditional machine learning baselines by 12.3%. The findings indicate that incorporating interaction metadata significantly improves emotion recognition performance, particularly for ambiguous or context-dependent cases. This research contributes a replicable framework for multimodal customer emotion analysis and offers practical implications for enhancing social media customer support systems through AI-driven sentiment intelligence.

• pdf

Copyright (c) 2026 Abey Litty (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.