GloVe and FastText in NLP

GloVe and FastText in NLP 

GloVe (Global Vectors) and FastText are advanced word embedding techniques used in Natural Language Processing (NLP) for generating dense vector representations of words. These models improve semantic understanding compared to traditional techniques such as Bag of Words and TF-IDF. 

Both GloVe and FastText are extensions of distributed word representation methods and are widely used in: 

Although both techniques generate dense embeddings, they differ in their learning approaches and handling of vocabulary 

GloVe (Global Vectors for Word Representation) is a word embedding model developed by Stanford University. 

GloVe combines: 

Unlike Word2Vec, which learns embeddings from local context windows, GloVe uses a global word co-occurrence matrix to learn relationships between words. 

Co-occurrence? 

Co-occurrence refers to how frequently two words appear together in a corpus. 

Example sentence: 

"The cat sits on the mat" 

Co-occurrence examples: 

GloVe learns semantic relationships from these co-occurrence statistics. 

Working Principle of GloVe 

The main idea behind GloVe is: Words that occur in similar contexts should have similar vector representations. 

Steps involved: 

Example Co-occurrence Matrix 

This matrix helps GloVe understand semantic relationships. 

GloVe Objective Function 

The GloVe equation is: The model minimizes differences between predicted and actual co-occurrence probabilities. 

Example Limitation 

Sentence 1: 

"I deposited money in the bank" 

Sentence 2: 

"The river bank is beautiful" 

GloVe generates the same embedding for “bank” in both sentences. 

FastText 

FastText is a word embedding model developed by Facebook AI Research. 

Unlike Word2Vec and GloVe, FastText represents words using: 

This enables FastText to handle: 

Working Principle of FastText 

FastText breaks words into smaller character-level units called n-grams. 

Example: 

Word: 

"playing" 

Character n-grams: 

["pla", "lay", "ayi", "yin", "ing"] 

The final word embedding is obtained by combining subword embeddings. 

Why FastText is Important 

Traditional embeddings fail for unknown words. 

Example: 

"computerization" 

Even if the exact word is unseen, FastText can infer meaning using subwords: 

["comp", "puter", "ization"] 

Thus, FastText reduces the Out-of-Vocabulary (OOV) problem. 

FastText Objective Function 

FastText extends Word2Vec using subword vectors. 

The prediction function is: 

Practical Implementation Using GloVe 

Install Libraries 

``` 

Hide 

pip install gensim 

``` 

GloVe Embeddings 

``` 

#Import libraries 

from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity 

#Sample documents 

documents = [ 

"king queen prince royal", 
 
"man woman boy girl", 
 
"apple banana mango fruit", 
 
"car bus train vehicle" 
 

] 

#Create TF-IDF Vectorizer 

vectorizer = TfidfVectorizer() 

#Convert text into vectors 

X = vectorizer.fit_transform(documents) 

#Display vocabulary 

print("Vocabulary:\n") 

print(vectorizer.get_feature_names_out()) 

#Convert sparse matrix to array 

vectors = X.toarray() 

#Display vectors 

print("\nTF-IDF Vectors:\n") 

print(vectors) 

#Calculate cosine similarity 

similarity = cosine_similarity(vectors) 

#Display similarity matrix 

print("\nCosine Similarity Matrix:\n") 

print(similarity) 

``` 

Implementation of FastText 

``` 

#Import library 

from gensim.models import FastText 

#Sample sentences 

sentences = [ 

["i", "love", "nlp"], 
 
["machine", "learning", "is", "powerful"], 
 
["deep", "learning", "improves", "ai"], 
 
["nlp", "uses", "machine", "learning"] 
 

] 

#Train FastText model 

model = FastText( 

sentences, 
 
vector_size=100, 
 
window=5, 
 
min_count=1, 
 
workers=4 
 

) 

#Display vector 

print("Word Vector for 'learning':\n") 

print(model.wv['learning']) 

#Similar words 

print("\nMost Similar Words:\n") 

print(model.wv.most_similar('learning')) 

``` 

Conclusion 

GloVe and FastText are highly influential word embedding techniques in Natural Language Processing. GloVe captures semantic relationships using global co-occurrence statistics, whereas FastText improves word representation using character-level subword information. Both methods provide dense semantic embeddings that significantly improve NLP tasks such as text classification, search systems, recommendation engines, and semantic analysis. Although modern contextual embeddings such as BERT and GPT provide superior performance, GloVe and FastText remain foundational techniques for understanding word embedding methodologies in NLP.