Coverting The Text to Vector(one hot encoding and bag of words method)

Nlp tutorial · PySpark.in

One-Hot Encoding in NLP

 Why do we need it?

• After text pre-processing (tokenization, stopword removal, stemming, lemmatization), text is still in string format.

• ML/DL models need numerical representation.

• One-hot encoding is the simplest way to represent words as vectors.

 Example Corpus

D1 = "The food is good"
D2 = "The food is bad"
D3 = "Pizza is amazing"

Step 1: Find Vocabulary
Unique words across all documents:
["the", "food", "is", "good", "bad", "pizza", "amazing"]

Vocabulary size (V) = 7

Step 2: Create One-Hot Vectors

Each word → represented by a vector of size V (vocabulary size).
Only the index of that word = 1, all others = 0.

Example:

• "the" → [1, 0, 0, 0, 0, 0, 0]

• "food" → [0, 1, 0, 0, 0, 0, 0]

• "is" → [0, 0, 1, 0, 0, 0, 0]

• "good" → [0, 0, 0, 1, 0, 0, 0]

• "bad" → [0, 0, 0, 0, 1, 0, 0]

• "pizza" → [0, 0, 0, 0, 0, 1, 0]

• "amazing" → [0, 0, 0, 0, 0, 0, 1]

 Step 3: Represent Documents

• D1 = "The food is good"

[[1,0,0,0,0,0,0], # the [0,1,0,0,0,0,0], # food [0,0,1,0,0,0,0], # is [0,0,0,1,0,0,0]] # good

Shape = (4 words × 7 vocab size) → (4×7)

• D2 = "The food is bad"

[[1,0,0,0,0,0,0], # the [0,1,0,0,0,0,0], # food [0,0,1,0,0,0,0], # is [0,0,0,0,1,0,0]] # bad

Shape = (4×7)

• D3 = "Pizza is amazing"

[[0,0,0,0,0,1,0], # pizza [0,0,1,0,0,0,0], # is [0,0,0,0,0,0,1]] # amazing

Shape = (3×7)

 Note:

One-hot encoding gives a unique binary vector to each word.
 The dimension of each vector = Vocabulary size.
 Each document becomes a matrix: (#words × vocab size).

Limitations (Why not used in NLP anymore?)

• Very high dimensional (huge vocab → very long sparse vectors).

• No semantic meaning (e.g., "good" and "bad" are equally far apart).

• Memory inefficient.

• Cannot capture context (word meaning depends on sentence).

Advantages of One Hot Encoding

1. Easy to implement

   • Libraries like sklearn (OneHotEncoder) and pandas.get_dummies() make it straightforward.

2. Clear and simple representation

   • Each word/token gets a unique binary vector, which avoids ambiguity.

 Disadvantages of One Hot Encoding

1. Sparse Matrix Problem

   • Results in high-dimensional vectors with mostly 0s.

   • Increases memory usage & computation.

   • Often leads to overfitting in ML models.

2. Variable Length Issues

   • Sentences can have different lengths (e.g., 4×7, 3×7 matrices).

   • ML models usually need fixed-size input vectors.

   • Padding or truncation becomes necessary.

3. No Semantic Meaning

   • Only represents presence/absence of words.

   • Doesn’t capture relationships or similarity between words.

   • Example: “food”, “pizza”, “burger” are equidistant in OHE space, though semantically related.

4. Out-of-Vocabulary (OOV) Problem

   • New/unseen words in test data cannot be represented if they weren’t in the training vocabulary.

   • This limits generalization.

5. High Dimensionality with Large Vocabularies

   • Real-world vocabularies can reach 50k+ unique words.

   • OHE vectors then become extremely large, inefficient, and redundant.

 Bag of Words (BoW) in NLP

 How it works:

1. Collect the dataset
   Example sentences:

   • S1: He is a good boy

   • S2: She is a good girl

   • S3: Boy and girl are good

2. Preprocessing

   • Convert all words to lowercase (avoid duplicates due to case).

   • Remove stopwords (he, she, is, a, and …).

   • Result:

     • S1 → good boy

     • S2 → good girl

     • S3 → boy girl good

3. Build Vocabulary
   Unique words after preprocessing:
   good, boy, girl

   • good → 3 times

   • boy → 2 times

   • girl → 2 times

    Vocabulary size = 3.

4. Create Feature Vectors (Bag of Words matrix)

   • Features = [good, boy, girl]

   • S1: good boy → [1, 1, 0]

   • S2: good girl → [1, 0, 1]

   • S3: boy girl good → [1, 1, 1]

   Each sentence is now a fixed-length vector.

5. Binary vs Frequency BoW

   • Binary BoW → if word present → 1, else 0.

   • Frequency BoW → word count is stored.

     • Example: “good good girl” → [2, 0, 1]

 Why Bag of Words is Useful

• Converts text into numerical vectors, so ML models can use it.

• Works well for text classification tasks like:

  • Spam vs Ham detection

  • Sentiment Analysis

  • Document categorization

 Key Difference from One-Hot Encoding

• OHE: Each word becomes its own vector → results in variable sentence sizes & very sparse matrices.

• BoW: Each sentence/document becomes a vector (fixed length = vocab size). In short

BoW solves the variable input size problem from One Hot Encoding by producing fixed-size sentence vectors, but still has some issues (which you’ll see in the next part: advantages vs disadvantages).

Why OHE is Rarely Used in NLP

• Works for very small, toy problems.

• Fails to handle real-world complexities (semantics, large vocabularies, efficiency).

• Modern alternatives: Bag of Words, TF-IDF, Word2Vec, GloVe, BERT embeddings.

Bag of Words – Advantages & Disadvantages

Advantages

1. Simple and intuitive

   • Very easy to understand and implement (e.g., CountVectorizer in sklearn).

2. Fixed-size input for ML algorithms

   • Unlike One-Hot Encoding, BoW produces fixed-length vectors (length = vocab size).

   • This makes it compatible with most ML models.

3. Captures frequency (optionally)

   • Unlike OHE (just 0/1), BoW can record how many times a word appears in a sentence.

Disadvantages

1. Sparse matrix problem

   • For large vocabularies (e.g., 50k words), each document vector will be extremely large with mostly 0s.

   • Leads to memory inefficiency + can cause overfitting.

2. Ignores word order (loss of context)

   • Sentences like:

     • “boy good girl” → [1,1,1]

     • “good girl boy” → [1,1,1]

   • Both produce the same vector, but meaning can differ → semantic info lost.

3. Out of Vocabulary (OOV) problem

   • If a new word (e.g., school) appears in test data but wasn’t in training vocab → it gets ignored.

   • Important context can be lost.

4. No true semantic meaning captured

   • Words are treated as independent tokens → no relationship between “good” vs “bad” or “food” vs “pizza”.

   • Example:

     • “The food is good” → [1,1,1,0]

     • “The food is not good” → [1,1,1,1]
       Cosine similarity might say they are similar, but meaning is opposite.

5. High dimensionality

   • Larger vocabularies = larger vectors = more storage & computation cost.

Comparison with One-Hot Encoding

• Solved problems:

  • Fixed-size vectors (BoW > OHE).

  • Represents entire sentence instead of single words.

• Still unsolved:

  • Sparse matrix

  • Out of vocabulary

  • Semantic meaning & context loss. In short

BoW is better than One-Hot Encoding because it produces fixed-size vectors and allows frequency counts, but it still suffers from sparsity, no semantics, OOV issues, and order loss. These gaps are what TF-IDF and Word Embeddings (Word2Vec, GloVe, embeddings in DL) try to fix.

More Nlp tutorials

• Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
• Tokenization in NLP and NLP Project Life Cycle
• Convert text to vector: N-grams and TF-IDF method
• Word Embedding
• What is Natural Language Processing ?
• Working with Text in NLP

All tutorials · Try the free PySpark compiler · Practice challenges