LLM Security & Red Teaming
Learn how LLM apps get attacked, from prompt injection and jailbreaks to data extraction and poisoning, then test and defend your own.
Before this: AI Engineering. AI Agents & MCP helps for the agent and tool-use steps.
-
25 min
Step 1The OWASP Top 10 for LLMs
A map of the main ways LLM applications get attacked.
OWASP's Top 10 Ways to Attack LLMs: AI Vulnerabilities ExposedIBM Technology
-
12 min
Step 2Prompt injection
Simon Willison on why mixing instructions and untrusted text is unsafe.
Prompt Injection, explainedSimon Willison
-
12 min
Step 3Jailbreaks
Computerphile on how prompts talk models past their safety training.
ChatGPT Jailbreak - ComputerphileComputerphile
-
31 min
Step 4Automated adversarial suffixes
Zico Kolter on optimized suffixes that jailbreak many aligned models.
Zico Kolter - Adversarial Attacks on Aligned Language ModelsFAR․AI
-
49 min
Step 5Indirect prompt injection in the wild
Real exploits where web pages and documents hijack LLM apps.
Indirect Prompt Injections in the Wild – Real World exploits and mitigations Johann RehbergerEkoparty Security Conference
-
11 min
Step 6Extracting training data
USENIX talk showing GPT-2 can regurgitate memorized training text.
USENIX Security '21 - Extracting Training Data from Large Language ModelsUSENIX
-
12 min
Step 7Backdoors and sleeper agents
Anthropic's study of hidden trigger behavior that survived safety training.
AI Sleeper Agents: How Anthropic Trains and Catches ThemRational Animations
-
34 min
Step 8Design patterns for secure agents
Limit what an agent can do after it reads untrusted input.
LLM Chronicles #6.9: Design Patterns for Securing LLM Agents Against Prompt Injection (Paper Review)Donato Capitella
-
21 min
Step 9Guardrails
Add input and output rails to a chatbot with NeMo Guardrails.
NVIDIA NeMo Guardrails: Full Walkthrough for Chatbots / AIJames Briggs
-
25 min
Step 10Automated red teaming with PyRIT
Microsoft's open tool for sending attack prompts and scoring replies.
BlueHat 2024: S24: Automate AI Red Teaming in your existing tool chain with PyRITMicrosoft Security Response Center (MSRC)
microsoft/PyRIT★ 5k -
19 min
Step 11Scan a model with garak
Run garak's probes against your own chatbot and read the report.
LLM Vulnerability Scanning with garak. Tutorial: Test your own chat bots!Embrace The Red
NVIDIA/garak★ 10.1k -
1h 15m
Step 12Defending against jailbreaks
Anthropic researchers on classifier defenses and their red-teaming results.
Defending against AI jailbreaksAnthropic
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Build a small RAG chatbot, plant an indirect prompt injection in one document, and show the attack and a fix.
- Run garak against a local open model and summarize which probe families succeeded.
- Add an input and output safety filter to a chatbot and measure how many of 50 jailbreak prompts it blocks.