Convert text to vector: N-grams and TF-IDF method

Nlp tutorial · PySpark.in

What are N-grams?

• N-grams = sequence of N words (contiguous).

• Helps capture context + word order.

• Example:

  • Bigram (n=2): “food good”, “not good”

  • Trigram (n=3): “food not good”

Applying Bigrams

Vocabulary (extended with bigrams):

• food, good, not, food good, food not, not good

Vectors:

• Sentence 1 ("food good") → [1, 1, 0, 1, 0, 0]

• Sentence 2 ("food not good") → [1, 1, 1, 0, 1, 1]

 Now the vectors are more different, so ML models can distinguish them better.

Trigrams (3-word combinations)

• Sentence 1: good boy → only 2 words → No trigram

• Sentence 2: good girl → only 2 words → No trigram

• Sentence 3: boy girl good → ["boy girl good"]

All possible trigrams in vocabulary:
["boy girl good"]

 Advantages of N-grams

1. Captures word order & context (unlike plain Bag of Words).

2. Helps detect semantic differences (e.g., “good” vs. “not good”).

3. Reduces the risk of treating opposite sentences as similar.

Limitations

1. Vocabulary explosion – more n-grams → higher dimensionality.

2. Still doesn’t fully capture meaning (semantics) like word embeddings (Word2Vec, GloVe). In short:

• Bag of Words treats words independently.

• N-grams add context by grouping words, making ML models smarter in detecting sentence meaning.

TF–IDF (Term Frequency – Inverse Document Frequency)

TF–IDF is a method to convert text into numerical vectors that give importance to words based on how often they appear in a document and how unique they are across documents.

 1. Components

(a) Term Frequency (TF)

Measures how often a word appears in a document.

T F ( t , d ) = Number of times word t appears in document d Total words in document d TF(t,d) = \frac{\text{Number of times word t appears in document d}}{\text{Total words in document d}}

(b) Inverse Document Frequency (IDF)

Measures how rare a word is across all documents.

I D F ( t ) = log ⁡ Total number of documents Number of documents containing word t IDF(t) = \log \frac{\text{Total number of documents}}{\text{Number of documents containing word t}}

• Common words (like "good" if present in all docs) get low weight.

• Rare but meaningful words (like "school") get high weight.

(c) TF–IDF Score

T F – I D F ( t , d ) = T F ( t , d ) × I D F ( t )

This balances local importance (inside a document) and global rarity (across documents).

 2. Example 

Sentences after preprocessing (lowercased, stopwords removed):

1. good boy

2. good girl

3. boy girl good

Step 1: Vocabulary

[good, boy, girl]

Step 2: Term Frequency (TF)

Step 3: Inverse Document Frequency (IDF)

• good → in all 3 docs →
  $\log (3/3)=\log (1)=0$

• boy → in 2 docs →
  $\log (3/2)\approx 0.176$

• girl → in 2 docs →
  $\log (3/2)\approx 0.176$

Step 4: TF–IDF (TF × IDF)

Final Sentence Vectors

• S1 ("good boy") → [0, 0.088, 0]

• S2 ("good girl") → [0, 0, 0.088]

• S3 ("boy girl good") → [0, 0.059, 0.059]

Why TF–IDF is Better than Bag of Words

 Assigns low weight to common words (e.g., “good”)
 Assigns high weight to rare words (more meaningful)
 Reduces the impact of stopwords / frequent words
 Creates more informative vectors for ML models

 Still, TF–IDF doesn’t capture word meaning or sequence (solved by embeddings like Word2Vec, GloVe, BERT). So in short:

• Bag of Words → counts word frequency (all words equal).

• TF–IDF → weights words by importance (rare > common).

TF–IDF:

 Advantages

1. Intuitive & easy to implement

   • Works similarly to Bag of Words in terms of vectorization.

   • Input size is fixed, based on vocabulary.

2. Fixed-size input representation

   • Like Bag of Words, vector length depends only on vocabulary size, not on sentence length.

3. Captures word importance 

   • Unlike Bag of Words (where all words are equal, just 0/1 or counts), TF–IDF gives different weights to words.

   • Example:

     • Sentences:

       1. good boy

       2. good girl

       3. boy girl good

     • Word good is present in all sentences → IDF = 0 → weight = 0 → considered unimportant.

     • Word boy appears in only some sentences → higher weight → considered important.

     • Word girl is important in its specific context (sentence 2, sentence 3).

   This ensures that rare, meaningful words are emphasized, while common words are ignored.
    Models trained on TF–IDF vectors get better accuracy than with Bag of Words.

 Disadvantages

1. Sparsity still exists

   • Many values are still zero, because most words don’t appear in every sentence.

   • High-dimensional sparse vectors → harder for some ML models.

2. OOV (Out-of-Vocabulary) problem

   • Vocabulary is fixed from training data.

   • If a new word appears in test data (not seen in training), it gets ignored (vector position = 0).

 Key Takeaway

• Bag of Words: Counts words, but all words have equal importance.

• TF–IDF: Counts words + gives importance weights (rare words → higher value, common words → lower value).

•  TF–IDF is almost always better than Bag of Words for text classification, clustering, and retrieval tasks.

•  But still limited by sparsity & OOV → later solved by Word Embeddings (Word2Vec, GloVe, BERT).

More Nlp tutorials

• Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
• Tokenization in NLP and NLP Project Life Cycle
• Coverting The Text to Vector(one hot encoding and bag of words method)
• Word Embedding
• What is Natural Language Processing ?
• Working with Text in NLP

All tutorials · Try the free PySpark compiler · Practice challenges