Multimodal Embeddings: Introduction & Use Cases (with Python)

Get exclusive access to AI resources and project ideas: https://the-data-entrepreneurs.kit.com/shaw Multimodal embeddings represent multiple data modalities in the same vector space. Here, I discuss how they are developed and two example use cases: 0-shot classification and image search. Resources: 📰 Blog: https://medium.com/towards-data-science/multimodal-embeddings-an-introduction-5dc36975966f?sk=8b7b6b81b3e890192aafeda15492c7de 💻 GitHub Repo: https://github.com/ShawhinT/YouTube-Blog/tree/main/multimodal-ai References: [1] BERT: https://arxiv.org/abs/1810.04805 [2] ViT: https://arxiv.org/abs/2010.11929 [3] CLIP: https://arxiv.org/abs/2103.00020 [4] Though2Text: https://arxiv.org/abs/2410.07507 [5] A Simple Framework for Contrastive Learning of Visual Representations: https://arxiv.org/abs/2002.05709 -- Homepage: https://www.shawhintalebi.com Introduction - 0:00 What are embeddings? - 1:01 Multimodal Embeddings - 5:08 Contrastive Learning - 6:56 Contrastive Learning (Details) - 8:16 Example 1: 0-shot Image Classification - 15:17 Example 2: Image Search - 19:50 What's Next? - 22:47