Multimodal & Vision-Language Models
How models see and talk: CLIP, SigLIP, Flamingo, BLIP-2, LLaVA-style models, and building a vision-language model yourself.
Before this: LLMs from Scratch. Computer Vision helps.
-
7 min
Step 1How multimodal models work
A plain overview of joining text, images, and audio.
How do Multimodal AI models work? Simple explanationAssemblyAI
-
22 min
Step 2Vision Transformers
Cut an image into patches and run a transformer.
Vision Transformers - Explained!CodeEmporium
-
7 min
Step 3CLIP
Learn a shared space for images and captions.
What CLIP models are (Contrastive Language-Image Pre-training)Super Data Science: ML & AI Podcast with Jon Krohn
mlfoundations/open_clip★ 14.3k -
25 min
Step 4Multimodal embeddings in Python
Search images with text and text with images.
Multimodal Embeddings: Introduction & Use Cases (with Python)Shaw Talebi
ShawhinT/YouTube-Blog★ 901 -
1h 00m
Step 5SigLIP
A simpler loss that scales contrastive training.
Lucas Beyer - Sigmoid Loss for Language Image Pre-TrainingCohere
-
10 min
Step 6What vision language models are
The standard recipe: vision encoder, adapter, language model.
What Are Vision Language Models? How AI Sees & Understands ImagesIBM Technology
-
13 min
Step 7Flamingo
Few-shot learning over mixed image and text prompts.
DeepMind Flamingo explained - 32 images are enoughBoris Meinardus
-
43 min
Step 8BLIP-2
Bridge a frozen image encoder and a frozen LLM.
Computer Vision Study Group Session on BLIP-2Hugging Face
-
46 min
Step 9Fine-tune a VLM with LoRA
Adapt a vision-language model to your data on one GPU.
Fine-Tune Visual Language Models (VLMs) - HuggingFace, PyTorch, LoRA, Quantization, TRLUygar Kurt
uygarkurt/Fine-Tune-VLMs★ 28 -
1h 10m
Step 10Vision-language models in CS231N
A university lecture on the current VLM landscape.
Stanford CS231N Deep Learning for Computer Vision | Spring 2025 | Lecture 16: Vision and LanguageStanford Online
-
21 min
Step 11Fine-tuning PaliGemma
Adapt a compact open VLM to a custom task.
Mastering Google's VLM PaliGemma: Tips And Tricks For Success and Fine TuningSam Witteveen
-
5h 46m
Step 12Code a vision language model from scratch
Implement PaliGemma end to end in PyTorch.
Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanationUmar Jamil
hkproj/pytorch-paligemma★ 637
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Build a text-to-image search app over your own photos with CLIP embeddings.
- Fine-tune a small open VLM on a custom visual question set and measure accuracy before and after.
- Write a short comparison of three open VLMs on ten of your own images.