Tokenization in NLP and NLP Project Life Cycle

Nlp tutorial · PySpark.in

What is Tokenization?

Tokenization is the process of breaking text (paragraphs, sentences) into smaller units called tokens.

• Sentence Tokenization → splitting a paragraph into sentences.

• Word Tokenization → splitting sentences into individual words.

These tokens form the vocabulary of your dataset.

 Example

 Text:

I like to drink apple juice. My friend likes mango juice.

Step 1: Sentence Tokenization

• Sentence 1 → "I like to drink apple juice."

• Sentence 2 → "My friend likes mango juice."

Step 2: Word Tokenization

Sentence 1 → ["I", "like", "to", "drink", "apple", "juice"]
Sentence 2 → ["My", "friend", "likes", "mango", "juice"]

 Vocabulary

• Total words = 11

• Unique words = 10 (because "juice" repeats)

• Vocabulary = {I, like, to, drink, apple, juice, My, friend, likes, mango}

Note: "like" and "likes" are treated as different words unless we apply stemming/lemmatization.

 Key Definitions

• Corpus → The entire collection of text (all documents).

• Document → A single paragraph or sentence from the corpus.

• Vocabulary → The set of all unique words from the corpus.

• Token → Each individual word or sentence after tokenization.

 NLP Project Life Cycle (Example: Sentiment Analysis)-

Step 1: Dataset

• Start with a dataset containing documents/sentences (D1, D2, …, corpus).

• Corpus → Collection of documents.

• Vocabulary → Unique set of words.

Step 2: Text Pre-processing 

Cleaning raw text before feeding it into ML/DL models.

• Tokenization: Split text into sentences/words.

• Lowercasing: Convert all text to lowercase (so Good and good are treated the same).

• Regular Expressions (Regex): Remove special characters, punctuation, numbers, etc.

Step 3: Text Pre-processing 

Further refining of tokens.

• Stemming: Reduce words to their root form (playing → play).

• Lemmatization: Similar but more accurate, uses dictionary (better → good).

• Stopwords removal: Remove common but non-informative words (is, the, and, etc.).

Step 4: Text → Vectors (Numerical Representation)

We can’t feed raw text to ML models, so convert words/sentences into vectors.
Techniques:

1. One Hot Encoding – simple but inefficient.

2. Bag of Words (BoW) – count word occurrences.

3. TF-IDF (Term Frequency – Inverse Document Frequency) – weighs words based on importance.

4. Word2Vec – dense word embeddings, captures semantic meaning.

5. Average Word2Vec – averages vectors of words in a sentence/document.

6. Advanced embeddings (later) – GloVe, FastText, Transformers (BERT, GPT, etc.).

Step 5: Model Training

• Use the vectors as input to Machine Learning models:

  • Logistic Regression, Naïve Bayes, SVM, Random Forest, etc.

• Or in Deep Learning:

  • RNN, LSTM, GRU, Transformers.

Step 6: Prediction & Evaluation

• Predict sentiment (positive/negative/neutral).

• Evaluate performance with Accuracy, Precision, Recall, F1-score.

More Nlp tutorials

• Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
• Coverting The Text to Vector(one hot encoding and bag of words method)
• Convert text to vector: N-grams and TF-IDF method
• Word Embedding
• What is Natural Language Processing ?
• Working with Text in NLP

All tutorials · Try the free PySpark compiler · Practice challenges