ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training

Add to your list(s) Download to your calendar using vCal

• Antonio Norelli, Sapienza University of Rome
• Wednesday 15 February 2023, 15:00-16:00
• Venue to be confirmed.

If you have a question about this talk, please contact Pietro Barbiero.

CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this talk, we will present the ASIF construction, showing that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs.

Then, we will discuss the unique properties of ASIF . Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. We will look at experiments on standard zero-shot visual benchmarks that demonstrate the typical transfer ability of image-text models. Overall, ASIF represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning.

This talk is part of the ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training series.

This talk is included in these lists:

• ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training
• Venue to be confirmed

Note that ex-directory lists are not shown.