Subword Models in NLP (BPE & WordPiece)

Subword Models are advanced tokenization techniques in Natural Language Processing (NLP) used to divide words into smaller meaningful units called subwords. These methods are extremely important in modern NLP systems because they help handle: 

Subword tokenization is widely used in: 

The two most important subword tokenization methods are: 

These techniques form the backbone of modern transformer-based NLP architectures. 

Need for Subword Models 

Traditional word-level tokenization has several problems. 

Example: 

"unhappiness" 

If this word is not present in the vocabulary: 

This creates the: 

Out-of-Vocabulary (OOV) Problem 

Subword models solve this issue by splitting words into smaller components. 

Example 

Word: 

"unhappiness" 

Subwords: 

["un", "happi", "ness"] 

Now the model can understand the word using known subwords. 

Byte Pair Encoding (BPE) 

BPE (Byte Pair Encoding) is one of the most widely used subword tokenization algorithms. 

Originally developed for data compression, BPE was later adapted for NLP. The algorithm attempts to maximize compression by merging frequent subword units. 

BPE works by: 

Example of BPE 

Suppose the corpus contains: 

low 
lowest 
newer 
wider 

Initial tokens: 

l o w 
l o w e s t 
n e w e r 
w i d e r 

Frequent pairs are merged gradually. 

Example merges: 

l + o → lo 
 
lo + w → low 

Final tokens may become: 

["low", "est", "new", "er"] 

Working Principle of BPE 

Steps: 

WordPiece Tokenization 

WordPiece is another popular subword tokenization algorithm. 

It is primarily used in: 

WordPiece is similar to BPE but uses probability-based merging rather than simple frequency counts. 

Example of WordPiece 

Word: 

"playing" 

Tokenization: 

["play", "##ing"] 

The prefix: 

"##" 

indicates continuation of a word. 

Working Principle of WordPiece 

Steps: 

WordPiece maximizes language modeling probability. 

The likelihood formula is: 

Example of WordPiece Tokenization 

Sentence: 

"Transformers are powerful" 

Possible tokens: 

["transform", "##ers", "are", "powerful"] 

Why Subword Models are Important in Transformers 

Transformer models use fixed vocabularies. 

Without subword tokenization: 

Subword models: 

Practical Implementation Using BPE 

Install Libraries 

``` 

Hide  

pip install tokenizers 

``` 

Example: 

``` 

#Import library 

from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.pre_tokenizers import Whitespace from tokenizers.trainers import BpeTrainer 

Sample dataset 

sentences = [ 

"Natural Language Processing is powerful", 
 
"Transformers improve NLP systems", 
 
"Deep Learning helps AI" 
 

] 

Create tokenizer 

tokenizer = Tokenizer(BPE()) 

Pre-tokenization 

tokenizer.pre_tokenizer = Whitespace() 

Trainer 

trainer = BpeTrainer( 

vocab_size=50, 
 
special_tokens=["[UNK]"] 
 

) 

#Train tokenizer 

tokenizer.train_from_iterator( 

sentences, 
 
trainer=trainer 
 

) 

#Encode sentence 

output = tokenizer.encode( 

"Transformers are powerful" 
 

) 

#Print tokens 

print("Tokens:\n") 

print(output.tokens) 

``` 

Practical Implementation Using WordPiece 

Install Libraries 

``` 

hide 

pip install transformers 

``` 

Example: 

``` 

#Import tokenizer 

from transformers import BertTokenizer 

#Load tokenizer 

tokenizer = BertTokenizer.from_pretrained( 

'bert-base-uncased' 
 

) 

#Tokenize sentence 

tokens = tokenizer.tokenize( 

"Transformers are amazing" 
 

) 

#Display tokens 

print("WordPiece Tokens:\n") 

print(tokens) 

``` 

Conclusion 

Subword models such as BPE and WordPiece represent one of the most important innovations in modern Natural Language Processing. These techniques solve critical vocabulary limitations by splitting words into smaller semantic units, enabling efficient handling of rare and unknown words. BPE provides simple frequency-based tokenization widely used in GPT architectures, while WordPiece offers probabilistic segmentation optimized for transformer models such as BERT. Subword tokenization has become an essential component of modern large language models and advanced NLP systems.