Sparse Embeddings in NLP (BM25 & SPLADE)

Sparse Embeddings are advanced text representation techniques used in Natural Language Processing (NLP) and Information Retrieval (IR). Unlike dense embeddings, sparse embeddings represent text using high-dimensional vectors where most values are zero. 

Sparse embeddings are highly effective for: 

The two most important sparse retrieval methods are: 

These methods are widely used in modern AI retrieval systems and search architectures. 

Sparse embeddings are vector representations where: 

Example: 

Vocabulary: 

["AI", "machine", "learning", "football"] 

Sentence: 

"AI learning" 

Sparse vector: 

[1,0,1,0] 

Only relevant terms are activated. 

Why Sparse Embeddings are Important 

Sparse retrieval methods are extremely useful because: 

Modern AI systems often combine: 

for hybrid retrieval systems. 

BM25 (Best Matching 25) 

BM25 is one of the most important ranking algorithms used in search engines. 

It improves traditional TF-IDF ranking by considering: 

BM25 is widely used in: 

Working Principle of BM25 

BM25 ranks documents according to their relevance to a query. 

Steps: 

BM25 Formula 

Components of BM25 

Example of BM25 

Query: 

"machine learning" 

Documents: 

Relevant documents receive higher scores. 

SPLADE (Sparse Lexical and Expansion Model) 

SPLADE is a modern sparse retrieval model that combines: 

Unlike BM25, SPLADE uses neural networks to generate sparse semantic representations. 

Main Idea of SPLADE 

SPLADE expands queries and documents using transformer-based lexical weighting. 

This allows: 

Working Principle of SPLADE 

Steps: 

SPLADE Objective 

SPLADE uses transformer-based activation functions. 

Sparse activation formula: 

Sparse Retrieval in Modern AI Systems 

Modern Retrieval-Augmented Generation (RAG) systems use: 

Sparse embeddings improve: 

Hybrid Retrieval Example 

Query: 

"AI healthcare applications" 

BM25 retrieves: 

Dense retrieval retrieves: 

Hybrid retrieval combines both. 

Practical Implementation Using BM25 

Install Libraries 

``` 

Hide 

pip install rank-bm25 

``` 

Exmaple: 

``` 

#Import library 

from rank_bm25 import BM25Okapi 

#Sample documents 

documents = [ 

"machine learning improves ai", 
 
"football is a popular sport", 
 
"deep learning powers modern ai", 
 
"healthcare uses artificial intelligence" 
 

] 

#Tokenize documents 

tokenized_docs = [ 

doc.split(" ") 
 
for doc in documents 
 

] 

#Create BM25 model 

bm25 = BM25Okapi(tokenized_docs) 

#Query 

query = "machine learning ai".split(" ") 

Calculate scores 

scores = bm25.get_scores(query) 

#Display results 

print("BM25 Scores:\n") 

print(scores) 

``` 

Practical Implementation Using SPLADE 

Install Libraries 

``` 

pip install transformers torch 

``` 

Exmaple: 

``` 

#Import libraries 

from transformers import AutoTokenizer, AutoModel import torch 

#Load tokenizer and model 

tokenizer = AutoTokenizer.from_pretrained( 

"distilbert-base-uncased" 
 

) 

model = AutoModel.from_pretrained( 

"distilbert-base-uncased" 
 

) 

Sample text 

text = "Artificial Intelligence improves healthcare" 

#Tokenize input 

inputs = tokenizer( 

text, 
 
return_tensors="pt" 
 

) 

#Generate embeddings 

outputs = model(**inputs) 

#Last hidden states 

embeddings = outputs.last_hidden_state 

#Print shape 

print("Embedding Shape:\n") 

print(embeddings.shape) 

``` 

Sparse Retrieval in RAG Systems 

Retrieval-Augmented Generation systems retrieve relevant documents before generating responses. 

Sparse retrieval improves: 

This is widely used in: 

Conclusion 

Sparse embeddings are highly important for modern information retrieval and semantic search systems. Techniques such as BM25 provide efficient lexical ranking using statistical relevance scoring, while advanced neural sparse models such as SPLADE combine transformer-based semantic understanding with sparse retrieval efficiency. These methods play a crucial role in search engines, enterprise retrieval systems, Retrieval-Augmented Generation (RAG), and modern AI-powered information access systems.