Word Embedding

Nlp tutorial · PySpark.in

Word Embeddings (NLP)

In NLP, one of the most important tasks is to represent words in a way that machines can understand. Words themselves are symbolic, so to perform tasks like text classification, sentiment analysis, or machine translation, we need to convert words into numerical vectors.

What are Word Embeddings?

“In NLP, word embedding is a term used for the representation of words for text analysis, typically in the form of real-valued vectors that encode the meaning of words such that words closer in the vector space are expected to be similar in meaning.”

In simpler terms, word embeddings map words into a continuous vector space. Words with similar meanings are placed close together, while words with opposite or unrelated meanings are far apart.

Word2Vec is a neural network-based model that learns word associations from a large corpus of text. 

Instead of sparse vectors filled with 0s and 1s (like in BoW/TF-IDF), Word2Vec produces dense vectors (e.g., 300 dimensions). These vectors encode semantic meaning — so words with similar meanings are closer in vector space.

By plotting these vectors (using techniques like PCA for dimensionality reduction), we can visualize semantic relationships between words.

Types of Word Embedding Techniques

Word embeddings can broadly be divided into two categories:

1. Count/Frequency-Based Methods

These are simpler techniques that rely on word frequency in documents.

Limitations of these methods:

2. Deep Learning-Based Methods

To overcome the limitations of frequency-based methods, deep learning techniques were introduced. The most famous one is:

Word2Vec

Word2Vec comes in two architectures:

  1. CBOW (Continuous Bag of Words) – Predicts a word based on its surrounding context.

  2. Skip-Gram – Predicts the surrounding words given a target word.

Example:

Word2Vec models can be trained from scratch, but they require a huge dataset. Luckily, pre-trained Word2Vec models.

Why Word2Vec is Better

Compared to one-hot, BoW, and TF-IDF, Word2Vec:

How Does Word2Vec Work?

At the core, Word2Vec represents each word as a vector of real numbers, derived by analyzing word usage patterns in text.

For example, consider a vocabulary with:

boy, girl, king, queen, apple, mango

Each word gets represented as a vector across multiple hidden features such as:

 We don’t manually define these features the model learns them automatically. But for intuition:

This way, semantic relationships emerge naturally.

 Example: Word Arithmetic

One of the most fascinating aspects of Word2Vec is word vector arithmetic.

For example:

king - man + woman ≈ queen

This works because the vectors capture gender and royalty relationships simultaneously.

Measuring Word Similarity

To measure how close two word vectors are, we often use Cosine Similarity:

Cosine Similarity = cos ⁡ ( θ ) = A ⋅ B ∣ ∣ A ∣ ∣ ∣ ∣ B ∣ ∣

Example:

Architectures of Word2Vec

Word2Vec has two main training architectures:

  1. CBOW (Continuous Bag of Words)

    • Predicts a word given its surrounding context.

    • Faster and works well with smaller datasets.

  2. Skip-Gram

    • Predicts surrounding words given a target word.

    • Better at capturing rare words and complex contexts.

Advantages of Word2Vec

 Dense vector representation (no sparsity).
 Captures semantic meaning (context-aware).
 Enables arithmetic on words.
 Improves NLP tasks like sentiment analysis, translation, recommendations.
 Available in pre-trained models.

Applications of Word2Vec

Word2Vec CBOW Model Explained Step by Step

In our earlier discussion, we introduced Word2Vec and its two architectures:

Now, we’ll go deeper into CBOW, understand how the model is created, what inputs/outputs look like, and how it is trained using deep learning concepts like loss functions and optimizers.

Pre-requisites

Before diving in, you should be familiar with:

Step 1: The Corpus

Let’s take a small example corpus to understand CBOW:

I neuron company is related to data science

 In real-world applications, Word2Vec is trained on massive corpora. For example, Google’s pre-trained Word2Vec model is trained on 3 billion words.

Step 2: Window Size

Word2Vec works by considering a window of context words around a target (center) word.

Suppose we choose window size = 5.
That means:

Example with window = 5:

Input (context): I, neuron, company, related, to Output (target): is

We slide the window across the sentence, generating multiple (input → output) training pairs.

Step 3: Converting Words into Vectors

Neural networks cannot understand raw text. So we convert words into numerical vectors.

Here’s how:

  1. Build a vocabulary:

    ["I", "neuron", "company", "is", "related", "to", "data", "science"]

    → 8 unique words.

  2. Represent each word using One-Hot Encoding:

    • "I" → [1, 0, 0, 0, 0, 0, 0, 0]

    • "company" → [0, 0, 1, 0, 0, 0, 0, 0]

    • "is" → [0, 0, 0, 1, 0, 0, 0, 0]

Each word is a vector of length |Vocab| = 8.

Step 4: The CBOW Neural Network

The CBOW model is essentially a shallow neural network with:

  1. Input Layer

    • Takes one-hot vectors of context words.

    • In our example: 4 words → 4 one-hot vectors.

  2. Hidden Layer

    • Dense layer with N neurons (embedding size).

    • Example: Google’s model uses N = 300.

    • Our toy example may use N = 5.

    This hidden layer is where the word embeddings are learned.

  3. Output Layer

    • Softmax layer of size |Vocab|.

    • Predicts the probability distribution over all words.

    • The correct output is the center word (target).

Step 5: Training the Model

The training process looks like this:

  1. Input context words → One-hot vectors.

  2. Average their embeddings (from hidden layer).

  3. Pass through output layer (softmax).

  4. Compare predicted word distribution (ŷ) with true output (y).

  5. Compute loss (cross-entropy).

  6. Backpropagate to adjust weights (embedding vectors).

This process repeats for every training pair until the embeddings capture semantic meaning.

Step 6: Output Word Vectors

After training:

These vectors capture semantic relationships:

Word2Vec Skip-Gram Model Explained

 We have discussed CBOW (Continuous Bag of Words) and how it predicts the center word given its context.

Now let’s explore the second Word2Vec architecture – Skip-Gram, understand how it differs from CBOW, and when to use it.

What is Skip-Gram?

The Skip-Gram model works the opposite way of CBOW:

In simple terms:

Example Sentence

Let’s use the same sentence:

I neuron company is related to data science

Suppose we set window size = 5.

In CBOW:

In Skip-Gram:

So, the direction of prediction changes.

Neural Network Structure

Skip-Gram uses a shallow neural network (like CBOW). Let’s break it down:

  1. Input Layer

    • One-hot vector of the center word.

    • If the vocabulary size is V = 7, the input vector has 7 dimensions.

    • Example: "is" → [0, 0, 1, 0, 0, 0, 0]

  2. Hidden Layer

    • Dense layer with N neurons (embedding size).

    • Example: N = 5 for a toy model, or N = 300 in Google’s pre-trained Word2Vec.

    • Learns the word embeddings.

  3. Output Layer

    • Softmax layer of size V.

    • Predicts probability distribution for each context word.

    • Since window size = 5, we expect 5 context outputs.

Training Process

  1. Input: One-hot encoding of the center word.

  2. Multiply with weights → project to embedding space.

  3. Predict context words using softmax.

  4. Compare predicted distribution (ŷ) with actual output (y).

  5. Compute loss (cross-entropy).

  6. Apply backpropagation + optimizer (e.g., SGD, Adam).

  7. Repeat until loss is minimized.

After training, every word is represented as a dense vector (N dimensions).

Example: Word Vectors

Suppose embedding size N = 5. After training, Skip-Gram might learn:

These vectors capture semantic meaning. For example:

When to Use CBOW vs Skip-Gram

How to Improve Skip-Gram (or CBOW)

  1. Increase Training Data

    • More data → better accuracy.

  2. Increase Window Size

    • Larger context window captures richer word relationships.

  3. Increase Embedding Dimension

    • Instead of 5, use 100–300 dimensions.

    • Example: Google’s pre-trained Word2Vec uses 300-dimensional embeddings trained on 3 billion words from Google News.

Pre-Trained Word2Vec Models

Instead of training from scratch, you can use pre-trained embeddings.

 Example: The word "cricket" would be represented as a 300-dimensional vector, capturing its real-world semantic relationships.

Advantages of Word2Vec

1. Dense Vector Representation

Why this matters: Dense vectors improve training efficiency and help machine learning models generalize better.

2. Captures Semantic Meaning

Unlike BoW or TF-IDF, Word2Vec goes beyond raw frequency. It captures the semantic relationship between words.

 This makes it possible to measure similarity between words using metrics like cosine similarity.

3. Fixed-Dimension Embeddings

Why this matters: This reduces dimensionality drastically and makes training efficient, even with very large corpora.

4. Handles Out-of-Vocabulary (OOV) Better

Earlier models struggled with out-of-vocabulary words (words not seen during training).

Why this matters: Makes Word2Vec more practical in real-world applications where new words frequently appear.

5. Improves Performance in Downstream Tasks

By providing meaningful word embeddings, Word2Vec improves performance in multiple NLP tasks such as:

Problem with plain Word2Vec

 Solution → Average Word2Vec

 Advantages

  1. Fixed-length representation: No matter how many words in the sentence, you always get a single 300-D vector.

  2. Captures semantic meaning: Since Word2Vec embeddings already encode semantic similarity, averaging helps keep the “gist” of the sentence.

  3. Efficient for classification: You can directly use this averaged vector as input to ML models (e.g., Logistic Regression, SVM, Neural Networks).

 

More Nlp tutorials

All tutorials · Try the free PySpark compiler · Practice challenges