Sheaf-Based Diffusion for Multimodal Graph Learning

Add to your list(s) Download to your calendar using vCal

• Mar Gonzàlez i Català (Universitat Politècnica de Catalunya)
• Tuesday 13 May 2025, 16:00-16:45
• Lecture Theatre 2, Computer Laboratory, William Gates Building.

If you have a question about this talk, please contact Pietro Lio.

Multimodal Graph Learning (MGL) is an emerging area in machine learning that focuses on graphs whose nodes carry information from different modalities, such as text and image. A central challenge in MGL is integrating these heterogeneous data types, which are not directly comparable. Standard Graph Neural Networks (GNNs) struggle in multimodal contexts because they assume homogeneity in node features and tend to merge modalities too early, leading to the loss of valuable, modality-specific information. Existing solutions address this by processing each modality independently and fusing their predictions at the output level. However, recent studies show that these late-fusion strategies underperform compared to general-purpose GNNs. To address this limitation, we introduce MMSheaf, a family of sheaf-based neural network architectures that preserve modality separation before diffusion and introduce structured, learnable mechanisms for cross-modal interaction during message passing. As a first contribution, we show that Sheaf Neural Networks (SNNs) outperform standard GNNs like GCN or GAT on multimodal graphs, proving to be an appropriate tool for this context. Building on this insight, our MMSheaf architecture further improves performance by explicitly modeling cross-modal interactions. We evaluate MMSheaf on synthetic multimodal datasets where successful classification requires integrating modalities in a non-trivial way. Additional experiments on the real-world Ele-Fashion dataset showcase the model’s effectiveness in practical multimodal settings. Overall, our findings establish sheaf-based diffusion as a powerful and expressive framework for Multimodal Graph Learning. Future work will apply this approach to diverse domains such as biomedicine and recommender systems.

Meet link: meet.google.com/wtt-wydt-hfk

This talk is part of the Foundation AI series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Artificial Intelligence Research Group Talks (Computer Laboratory)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Department of Computer Science and Technology talks and seminars
• Guy Emerson's list
• Hanchen DaDaDash
• Interested Talks
• Lecture Theatre 2, Computer Laboratory, William Gates Building
• School of Technology
• Speech Seminars
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list
• yk449

Note that ex-directory lists are not shown.