Contextual Embeddings in NLP (ELMo, BERT, GPT)

Contextual Embeddings are one of the most advanced and important developments in Natural Language Processing (NLP). Unlike traditional embedding techniques such as Word2Vec, GloVe, and FastText, contextual embeddings generate different vector representations for the same word depending on its context in a sentence. 

This advancement significantly improved machine understanding of language semantics, ambiguity, and context. 

Contextual embedding models are widely used in: 

The most important contextual embedding models are: 

These models transformed modern NLP and became the foundation of Generative AI systems. 

Need for Contextual Embeddings 

Traditional embeddings generate only one vector per word. 

Example: 

"I deposited money in the bank" 
 
"The river bank is beautiful" 

The word: 

"bank" 

has two different meanings. 

However: 

This creates ambiguity. 

Contextual embeddings solve this problem by generating context-dependent vectors. 

What are Contextual Embeddings? 

Contextual embeddings generate dynamic vectors based on sentence context. 

Example: 

Contextual models assign different embeddings for each usage. 

Evolution of Word Embeddings 

ELMo (Embeddings from Language Models) 

ELMo was developed by AllenNLP in 2018. 

It introduced the idea of: 

Contextual word embeddings using deep bidirectional language models. 

ELMo uses: 

GPT (Generative Pre-trained Transformer) 

GPT was developed by OpenAI. 

GPT is a: 

Decoder-only transformer model designed for text generation. 

Unlike BERT: 

GPT predicts the next word sequentially. 

Example of GPT 

Input: 

"Artificial Intelligence is" 

Prediction: 

"transforming the world" 

GPT Objective Function 

Architecture of GPT 

GPT consists of: 

Limitations of GPT 

Practical Implementation Using BERT 

Install Libraries 

``` 

pip install transformers torch 

``` 

Example: 

``` 

Import libraries 

from transformers import BertTokenizer, BertModel import torch 

Load pretrained BERT tokenizer and model 

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') 

model = BertModel.from_pretrained('bert-base-uncased') 

Sample text 

text = "Natural Language Processing is amazing" 

Tokenize input 

inputs = tokenizer(text, return_tensors='pt') 

Generate embeddings 

outputs = model(**inputs) 

Last hidden states 

embeddings = outputs.last_hidden_state 

Print embedding shape 

print("Embedding Shape:\n") 

print(embeddings.shape) 

``` 

Explanation: 

Implementation Using GPT 

``` 

Import pipeline 

from transformers import pipeline 

Create text generator 

generator = pipeline( 

'text-generation', 
 
model='gpt2' 
 

) 

Generate text 

output = generator( 

"Artificial Intelligence is", 
 
max_length=30, 
 
num_return_sequences=1 
 

) 

Print generated text 

print(output[0]['generated_text']) 

``` 

Conclusion 

Contextual embeddings represent one of the most transformative advancements in Natural Language Processing. Models such as ELMo, BERT, and GPT introduced dynamic context-aware word representations that significantly improved machine understanding of language. ELMo pioneered bidirectional contextual embeddings using LSTMs, BERT revolutionized NLP using transformer encoders and bidirectional attention, and GPT enabled highly advanced generative language modeling. These models form the foundation of modern AI systems, including chatbots, search engines, virtual assistants, and large language models.