A Multimodal Deep Learning Approach Integrated with Clinically Constrained Counterfactual Explainable AI (CXAI) for Shared Decision-Makings

• Abey City

  Texas University

  Author

Thyroid nodule malignancy risk stratification remains a significant clinical challenge, with ultrasound-guided fine-needle aspiration (FNA) biopsies frequently yielding indeterminate cytology results (20% of cases) that complicate treatment decisions . While deep learning has shown promise in medical image classification, existing approaches predominantly rely on single-modality analysis, limiting diagnostic accuracy . Furthermore, the "black box" nature of AI systems impedes clinical adoption and shared decision-making between clinicians and patients . This study proposes and validates a multimodal deep learning framework that integrates B-mode ultrasound, strain elastography, and clinical text reports through a bidirectional cross-modal attention mechanism, enhanced by a novel Clinically Constrained Counterfactual Explainable AI (CXAI) module. The CXAI module generates clinically meaningful counterfactual explanations by systematically modifying key imaging and clinical features to demonstrate how diagnostic decisions would change under alternative evidence conditions . The framework was developed on 1,472 cases from Xi'an International Medical Center Hospital and externally validated on 4,530 cases across two clinical centers and public datasets (DDTI, TN3K). Our approach achieved an AUC of 0.937 (95% CI: 0.914–0.960) on internal validation and 0.896 (95% CI: 0.887–0.905) on external validation . The multimodal model significantly outperformed single-modal baseline models including ResNet-50 (AUC: 0.841), DenseNet-121 (AUC: 0.848), and Vision Transformer (AUC: 0.835) (all p < 0.001). The CXAI module generated explanations that correlated with clinically recognized biomarkers, enabling radiologists to verify diagnostic decisions and facilitating meaningful shared decision-making. This framework addresses critical barriers to AI adoption in thyroid care by combining superior predictive performance with transparent, clinically meaningful explanations.

• PDF

Copyright (c) 2026 Abey City (Author)

This work is licensed under a Creative Commons Attribution 4.0 International License.