Distributed Word Representations
Nlp tutorial · PySpark.in
Distributed Word Representations
Traditional methods fail to capture semantic similarity.
Modern NLP introduced:
Word embeddings are dense vector representations where semantically similar words are close in vector space.
Example:
vector("king") - vector("man") + vector("woman") ≈ vector("queen")
Properties of Word Embeddings
Property | Meaning |
Dense vectors | Small dimensional representations |
Semantic similarity | Similar words near each other |
Continuous space | Better generalization |
Learned automatically | Through neural networks |
Word2Vec
Developed by Google in 2013.Word2Vec learns word representations using shallow neural networks.
Two architectures:
- CBOW (Continuous Bag of Words)
- Skip-Gram
```
from gensim.models import Word2Vec
sentences = [
["king", "is", "a", "man"],
["queen", "is", "a", "woman"],
["man", "and", "woman", "are", "human"],
["king", "and", "queen", "rule"]
]
model = Word2Vec(sentences, vector_size=50, min_count=1)
# Similarity score
similarity = model.wv.similarity('king', 'queen')
print("Similarity between king and queen:")
print(similarity)
```
Word embeddings can measure semantic similarity.
```
from gensim.models import Word2Vec
sentences = [
["king", "is", "a", "man"],
["queen", "is", "a", "woman"],
["man", "and", "woman", "are", "human"],
["king", "and", "queen", "rule"]
]
model = Word2Vec(sentences, vector_size=50, min_count=1)
# Similarity score
similarity = model.wv.similarity('king', 'queen')
print("Similarity between king and queen:")
print(similarity)
```
Word2Vec supports vector arithmetic.
```
from gensim.models import Word2Vec
sentences = [
["king", "man", "royal"],
["queen", "woman", "royal"],
["man", "boy"],
["woman", "girl"]
]
model = Word2Vec(sentences, vector_size=50, min_count=1)
result = model.wv.most_similar(
positive=['king', 'woman'],
negative=['man'],
topn=1
)
print(result)
```
CBOW (Continuous Bag of Words)
Predicts the target word from surrounding context words.
Example:
Sentence:
“I love natural language processing”
Input:
["I", "love", "language", "processing"]
Output:
"natural"
Architecture
Context Words → Neural Network → Target Word
CBOW Training Example using Gensim
```
from gensim.models import Word2Vec
# Training corpus
sentences = [
["i", "love", "natural", "language", "processing"],
["nlp", "is", "very", "interesting"],
["machine", "learning", "uses", "nlp"],
["deep", "learning", "improves", "ai"]
]
# CBOW Model
# sg=0 means CBOW
model = Word2Vec(
sentences,
vector_size=100,
window=2,
min_count=1,
sg=0
)
# Word vector
print("Vector for 'nlp':")
print(model.wv['nlp'])
# Similar words
print("\nMost Similar Words:")
print(model.wv.most_similar('nlp'))
```
Advantages
- Faster training
- Good for frequent words
- Computationally efficient
Limitations
- Poor for rare words
- Less effective on small datasets
Skip-Gram
Predicts surrounding context words from a target word.
Example:
Input:
"natural"
Output:
["I", "love", "language", "processing"]
Architecture
Target Word → Neural Network → Context Words
```
from gensim.models import Word2Vec
sentences = [
["i", "love", "natural", "language", "processing"],
["nlp", "is", "very", "interesting"],
["machine", "learning", "uses", "nlp"],
["deep", "learning", "improves", "ai"]
]
# Skip-Gram Model
# sg=1 means Skip-Gram
model = Word2Vec(
sentences,
vector_size=100,
window=2,
min_count=1,
sg=1
)
# Word vector
print("Vector for 'learning':")
print(model.wv['learning'])
# Similar words
print("\nMost Similar Words:")
print(model.wv.most_similar('learning'))
```
Advantages
- Better semantic understanding
- Good for rare words
- Higher accuracy
Limitations
- Slower training
- Computationally expensive
CBOW vs Skip-Gram
Feature | CBOW | Skip-Gram |
Training Speed | Fast | Slow |
Rare Words | Weak | Strong |
Dataset Size | Small datasets | Large datasets |
Accuracy | Moderate | High |
Word2Vec Mathematical Objective
Skip-Gram maximizes: Probability(context words | target word)
```
from gensim.models import Word2Vec
sentences = [
["i", "love", "nlp"],
["nlp", "is", "powerful"],
["machine", "learning", "and", "nlp"]
]
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)
print(model.wv['nlp'])
print(model.wv.most_similar('nlp'))
```
GloVe (Global Vectors)
Developed by Stanford. Words appearing in similar contexts should have similar embeddings.
Combines:
- Matrix factorization
- Local context learning
Unlike Word2Vec, GloVe uses:
GloVe Training Example
```
from glove import Corpus, Glove
# Training data
sentences = [
["i", "love", "nlp"],
["nlp", "is", "powerful"],
["deep", "learning", "uses", "nlp"]
]
# Create corpus
corpus = Corpus()
corpus.fit(sentences, window=5)
# Train GloVe model
glove = Glove(
no_components=100,
learning_rate=0.05
)
glove.fit(
corpus.matrix,
epochs=30,
no_threads=4,
verbose=True
)
# Add dictionary
glove.add_dictionary(corpus.dictionary)
# Vector for word
print(glove.word_vectors[glove.dictionary['nlp']])
```
Global Co-occurrence Statistics
Co-occurrence Matrix
Counts how often words appear together.
Example:
Word | ice | steam |
solid | 20 | 1 |
gas | 1 | 18 |
This captures semantic relationships.
```
from collections import defaultdict
import pandas as pd
corpus = [
["ice", "solid"],
["steam", "gas"],
["ice", "cold"],
["steam", "hot"]
]
window_size = 1
co_occurrence = defaultdict(lambda: defaultdict(int))
for sentence in corpus:
for i, word in enumerate(sentence):
start = max(i - window_size, 0)
end = min(i + window_size + 1, len(sentence))
for j in range(start, end):
if i != j:
neighbor = sentence[j]
co_occurrence[word][neighbor] += 1
# Convert to DataFrame
df = pd.DataFrame(co_occurrence).fillna(0)
print(df)
```
Advantages
- Captures global statistics
- Better semantic relationships
- Efficient training
Limitations
- Static embeddings
- Same word has same meaning everywhere
FastText
Developed by Facebook. FastText improves Word2Vec by using:
```
from gensim.models import FastText
sentences = [
["i", "love", "nlp"],
["deep", "learning", "is", "powerful"],
["fasttext", "handles", "rare", "words"]
]
# Train FastText model
model = FastText(
sentences,
vector_size=100,
window=3,
min_count=1
)
# Word vector
print("Vector for 'nlp':")
print(model.wv['nlp'])
# Similar words
print("\nMost Similar:")
print(model.wv.most_similar('learning'))
```
FastText Unknown Word Handling
Major Advantage
FastText can generate embeddings for unseen words.
```
from gensim.models import FastText
sentences = [
["machine", "learning"],
["deep", "learning"]
]
model = FastText(
sentences,
vector_size=50,
min_count=1
)
# Unknown word
print(model.wv['learner'])
```
Character-level subword information
Example
Word:
"playing"
Subwords:
<pla, play, layi, ying>
Advantages
- Handles unknown words
- Good for morphologically rich languages
- Better rare word handling
FastText vs Word2Vec
Feature | Word2Vec | FastText |
OOV Words | Poor | Excellent |
Subword Modeling | No | Yes |
Rare Words | Weak | Strong |
```
from gensim.models import FastText
sentences = [
["i", "love", "nlp"],
["deep", "learning", "is", "powerful"]
]
model = FastText(sentences, vector_size=100, window=5, min_count=1)
print(model.wv['nlp'])
```
Real-World Applications
Application | Usage |
|---|---|
Machine Translation | Semantic understanding |
Chatbots | Intent detection |
Search Engines | Query matching |
Recommendation Systems | Similarity learning |
Sentiment Analysis | Feature representation |
Question Answering | Context understanding |
More Nlp tutorials
- Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
- Tokenization in NLP and NLP Project Life Cycle
- Coverting The Text to Vector(one hot encoding and bag of words method)
- Convert text to vector: N-grams and TF-IDF method
- Word Embedding
- What is Natural Language Processing ?
All tutorials · Try the free PySpark compiler · Practice challenges