Classic NLP Before LLMs
The NLP toolkit that came before transformers: tokenizing and normalizing text, TF-IDF, n-grams, Naive Bayes, HMM tagging, NER, topic models, word vectors, and seq2seq with attention.
Before this: Machine Learning. Deep Learning for the seq2seq and attention steps.
-
10 min
Step 1What NLP is
The main NLP tasks and how a text pipeline fits together.
What is NLP (Natural Language Processing)?IBM Technology
-
8 min
Step 2Text tokenization
Split raw text into tokens and see why the choices matter.
NLP Demystified 2: Text TokenizationFuture Mojo
-
17 min
Step 3Stemming and lemmatization
Reduce words to a shared base form with NLTK and spaCy.
Stemming and Lemmatization: NLP Tutorial For Beginners - S1 E10codebasics
codebasics/nlp-tutorials★ 341 -
9 min
Step 4TF-IDF
Weight word counts by how rare each word is across documents.
Term Frequency Inverse Document Frequency (TF-IDF) ExplainedDataMListic
-
11 min
Step 5N-gram language models
Predict the next word from counts of the previous few words.
NLP: Understanding the N-gram language modelsMachine Learning TV
-
15 min
Step 6Naive Bayes for spam filtering
A fast text classifier that assumes words are independent given the class.
Naive Bayes, Clearly Explained!!!StatQuest with Josh Starmer
-
45 min
Step 7Sentiment analysis: VADER to RoBERTa
Score reviews with a word-list method, then a pretrained model, and compare.
Python Sentiment Analysis Project with NLTK and 🤗 Transformers. Classify Amazon Reviews!!Rob Mulla
-
11 min
Step 8Part-of-speech tagging
Tag each word's grammatical role with a hidden Markov model.
Part of Speech Tagging : Natural Language Processingritvikmath
-
21 min
Step 9The Viterbi algorithm
Find the most likely tag sequence with dynamic programming.
The Viterbi Algorithm : Natural Language Processingritvikmath
-
23 min
Step 10Named entity recognition
Pull people, places, and organizations out of text with spaCy.
Named Entity Recognition (NER): NLP Tutorial For Beginners - S1 E12codebasics
explosion/spaCy★ 34.1k -
27 min
Step 11Topic modeling with LDA
Model each document as a mix of topics, and each topic as a mix of words.
Latent Dirichlet Allocation (Part 1 of 2)Luis Serrano Academy
piskvorky/gensim★ 16.5k -
16 min
Step 12Word embeddings and word2vec
Learn word vectors by predicting the words around each word.
Word Embedding and Word2Vec, Clearly Explained!!!StatQuest with Josh Starmer
-
13 min
Step 13GloVe and fastText
Word vectors from co-occurrence counts and from subword pieces.
Word2Vec, GloVe, FastText- EXPLAINED!CodeEmporium
-
17 min
Step 14Seq2seq encoder-decoder models
Map one sequence to another, as in translation, with two LSTMs.
Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks, Clearly Explained!!!StatQuest with Josh Starmer
bentrevett/pytorch-seq2seq★ 5.7k -
16 min
Step 15Attention for seq2seq
Let the decoder look back at every input word, the step toward transformers.
Attention for Neural Networks, Clearly Explained!!!StatQuest with Josh Starmer
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Build a spam filter with TF-IDF and Naive Bayes, and report precision and recall on a held-out set.
- Run LDA on a news or Reddit dataset with gensim and label the top 10 topics by their words.
- Train word2vec on a book from Project Gutenberg and show nearest neighbors and analogies.