Track

Multimodal

Goal: Connect vision and language with contrastive encoders, then generative VLMs (Flamingo, BLIP-2, LLaVA), and finally joint spaces across many modalities.

Prereqs: Deep Learning. Computer Vision and LLMs help.

Status: done

Work through the steps in order. Bold links open YouTube.

Step	Concept	YouTube	Read
1	Vision-language models	What Are Vision Language Models? How AI Sees & Understands Images	Lilian Weng — Generalized Visual Language Models
2	Contrastive language-image pretraining	Yannic Kilcher — OpenAI CLIP: Connecting Text and Images	Radford et al. 2021 — Learning Transferable Visual Models From Natural Language Supervision
3	Visual question answering	Visual Question Answering (VQA)	Antol et al. 2015 — VQA: Visual Question Answering
4	Few-shot visual language models	Flamingo: Visual Language Model for Few-Shot Learning	Alayrac et al. 2022 — Flamingo: a Visual Language Model for Few-Shot Learning
5	Bootstrapping VLMs with frozen encoders	BLIP-2 Architecture in 3 minutes!	Li et al. 2023 — BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
6	Visual instruction tuning	AI Bites — LLaVA: the first instruction following multi-modal model	Liu et al. 2023 — Visual Instruction Tuning
7	Large multimodal models landscape	CVPR 2023 Tutorial — Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4	Li et al. 2023 — Large Multimodal Models: Notes on CVPR 2023 Tutorial
8	Improved multimodal instruction baselines		Liu et al. 2023 — Improved Baselines with Visual Instruction Tuning
9	Sigmoid language-image pretraining	Lucas Beyer — Sigmoid Loss for Language Image Pre-Training	Zhai et al. 2023 — Sigmoid Loss for Language Image Pre-Training
10	Binding many modalities	AI Bites — ImageBind: One Embedding Space To Bind Them All	Girdhar et al. 2023 — ImageBind: One Embedding Space To Bind Them All
11	Instruction-aware vision-language models		Dai et al. 2023 — InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Found a broken link or an unclear step? Report a problem with this track.