AI Roadmaps
← All roadmaps
AdvancedRoadmap 19 · 12 steps · 11h 09m of video

Multimodal & Vision-Language Models

How models see and talk: CLIP, SigLIP, Flamingo, BLIP-2, LLaVA-style models, and building a vision-language model yourself.

Before this: LLMs from Scratch. Computer Vision helps.

  1. 7 min
    Step 1

    How multimodal models work

    A plain overview of joining text, images, and audio.

    How do Multimodal AI models work? Simple explanation

    AssemblyAI

  2. 22 min
    Step 2

    Vision Transformers

    Cut an image into patches and run a transformer.

    Vision Transformers - Explained!

    CodeEmporium

  3. 7 min
    Step 3

    CLIP

    Learn a shared space for images and captions.

    What CLIP models are (Contrastive Language-Image Pre-training)

    Super Data Science: ML & AI Podcast with Jon Krohn

    mlfoundations/open_clip★ 14.3k
  4. 25 min
    Step 4

    Multimodal embeddings in Python

    Search images with text and text with images.

    Multimodal Embeddings: Introduction & Use Cases (with Python)

    Shaw Talebi

    ShawhinT/YouTube-Blog★ 901
  5. 1h 00m
    Step 5

    SigLIP

    A simpler loss that scales contrastive training.

    Lucas Beyer - Sigmoid Loss for Language Image Pre-Training

    Cohere

  6. 10 min
    Step 6

    What vision language models are

    The standard recipe: vision encoder, adapter, language model.

    What Are Vision Language Models? How AI Sees & Understands Images

    IBM Technology

  7. 13 min
    Step 7

    Flamingo

    Few-shot learning over mixed image and text prompts.

    DeepMind Flamingo explained - 32 images are enough

    Boris Meinardus

  8. 43 min
    Step 8

    BLIP-2

    Bridge a frozen image encoder and a frozen LLM.

    Computer Vision Study Group Session on BLIP-2

    Hugging Face

  9. 46 min
    Step 9

    Fine-tune a VLM with LoRA

    Adapt a vision-language model to your data on one GPU.

    Fine-Tune Visual Language Models (VLMs) - HuggingFace, PyTorch, LoRA, Quantization, TRL

    Uygar Kurt

    uygarkurt/Fine-Tune-VLMs★ 28
  10. 1h 10m
    Step 10

    Vision-language models in CS231N

    A university lecture on the current VLM landscape.

    Stanford CS231N Deep Learning for Computer Vision | Spring 2025 | Lecture 16: Vision and Language

    Stanford Online

  11. 21 min
    Step 11

    Fine-tuning PaliGemma

    Adapt a compact open VLM to a custom task.

    Mastering Google's VLM PaliGemma: Tips And Tricks For Success and Fine Tuning

    Sam Witteveen

  12. 5h 46m
    Step 12

    Code a vision language model from scratch

    Implement PaliGemma end to end in PyTorch.

    Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanation

    Umar Jamil

    hkproj/pytorch-paligemma★ 637

Build it

Watching isn't learning. Pick one and put it on GitHub.

Next roadmap