Logo image
Back
Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention
Conference proceeding   Open access

Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

Joy Dhar, Manish Kumar Pandey, Nayyar Zaidi, Chen Chen, Maryam Haghighat, Ferdous Sohel and Puneet Goyal
Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pp.863-874
ACM Conferences
KDD '26: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Jeju Island Republic of Korea, 09/08/2026–13/08/2026)
2026
DOI: https://doi.org/10.1145/3770855.3817885
pdf
3770855.38178853.76 MBDownloadView
Open Access CC BY V4.0

Abstract

Medical Image Analysis Multimodal Fusion Attention
Multimodal fusion learning (MFL) (a framework to jointly learn from heterogeneous data sources) has shown great potential in various fields such as Medicine, Science, and Engineering. It is extremely desirable in the medical domain, where we are faced with disparate data modalities such as imaging, clinical records, and omics. However, existing MFL strategies face several major challenges. First, they struggle to capture complex cross-modal interactions effectively, which in turn limits performance improvements. Second, they incur high computational costs, restricting their applicability in resource-constrained healthcare AI applications. Finally, they are often designed and evaluated for narrow, fixed modality configurations (e.g., imaging-only, or specific pairs such as image and omics), which limits evidence of their adaptability and generalizability to broader collections of heterogeneous medical modalities. To address these challenges, we propose a novel MFL framework – Cascaded Unified Representation Learning for Efficient Fusion Network (CURE) – a lightweight and scalable framework that progressively integrates various modalities through a novel efficient Hybrid Geometry Aware Fusion layer (HyFuse), where each HyFuse layer is sequentially learned for each modality, making the framework adaptable and generalizable. Within HyFuse, an efficient residual convolution module captures rich multi-scale features to ensure cost-effective learning, while a hybrid-space aware attention mixer learns coarse-to-fine structural cues to better preserve cross-modal relationships. Complementary learnable late-fusion and shared-information refinement modules are then employed to learn robust, modality-order-invariant shared features, which in turn yields consistent performance improvements. Extensive evaluations on 16 public datasets show that CURE outperforms leading multimodal fusion methods (e.g., DRIFA-Net and HEALNet), boosting performance by up to ≈ 3.97% and lowering computational costs by up to ≈ 87.8%, ensuring more effective and reliable predictions.

Details

Metrics

1 Record Views
Logo image