Breaking Text into Meaning units
Nlp tutorial · PySpark.in
The Text Preprocessing Pipeline
• Understand why text preprocessing is required in NLP
• Learn the steps of the text preprocessing pipeline
• Implement preprocessing steps using Python
1. What Is Text Preprocessing?
Text preprocessing is the process of converting raw and unstructured text into a clean format that machines can understand. In real-world data, text is often messy. It may contain punctuation, mixed cases, and unnecessary words. So before applying any NLP model, preprocessing becomes essential.
2. Why Is Text Preprocessing Important?
Text preprocessing is important because:
- Raw text contains noise
- Noise affects model accuracy
- Clean text improves learning
For example, the words “Data”, “data”, and “DATA” look different to a machine.
But for humans, they mean the same thing.
Preprocessing removes this confusion and makes text consistent.
3. Text Preprocessing Pipeline
Steps include -
• Lowercasing
• Removing punctuation
• Tokenization
• Stopword removal
• Stemming / Lemmatization
4. Lowercasing
Lowercasing converts all characters in text to lowercase.
This step is simple but very important.
Why?
Because machines treat “NLP” and “nlp” as different words.
After lowercasing, such confusion is removed.
Example
```
text = "Natural Language Processing Is Powerful"
text = text.lower()
print(text)
```
5. Removing Punctuation
Punctuation marks like . , ! ? usually do not add meaning in most NLP tasks.
For example:
“Good job!”
“Good job”
Both convey the same meaning for machines.
Removing punctuation simplifies text processing.
Example
```
text = "NLP is powerful, and useful!"
clean_text = text.translate(str.maketrans("", "", string.punctuation))
print(clean_text)
```
6. Tokenization
Tokenization is the process of splitting text into smaller parts called tokens.
Usually, tokens are words.
Instead of working with a full sentence, the machine now works with individual words.
Python Example
```
text = "nlp makes machines understand language"
tokens = text.split()
print(tokens)
```
Types of Tokenization
1. Word Tokenization
Word tokenization splits text into individual words. It is the most commonly used type of tokenization.
Example:
Sentence:
“NLP makes machines understand language”
Tokens:
['NLP', 'makes', 'machines', 'understand', 'language']
2. Sentence Tokenization
Sentence tokenization splits a paragraph into individual sentences.
This is useful for document analysis and summarization.
Example:
Text:
“NLP is interesting. It is widely used in AI.”
Tokens:
['NLP is interesting.', 'It is widely used in AI.']
3. Character Tokenization
Character tokenization breaks text into individual characters.
Example:
Word:
“NLP”
Tokens:
['N', 'L', 'P']
4. Subword Tokenization
Subword tokenization splits words into smaller meaningful parts.
This is widely used in modern NLP models like BERT and GPT.
Example:
Word:
“unhappiness”
Tokens:
['un', 'happi', 'ness']
7. Stopword Removal
Stopwords are commonly used words like:
is, the, in, and, of
These words appear frequently but usually do not carry useful information.
Removing them:
- Reduces text size
- Improves model focus
Example
```
text = "nlp is used in many real world applications"
tokens = text.split()
stop_words = set(stopwords.words('english'))
filtered_words = [word for word in tokens if word not in stop_words]
print(filtered_words)
```
8. Stemming and Lemmatization
Stemming
Stemming reduces words to their root form by removing suffixes.
It is fast but sometimes inaccurate.
Examples:
- running → run
- studies → studi
```
stemmer = PorterStemmer()
print(stemmer.stem("running"))
```
Lemmatization
Lemmatization converts words to their correct base form using grammar rules.
It is slower but more accurate.
Examples:
- running → run
- better → good
```
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("running", pos="v"))
```
9. Complete Example
A full pipeline combines all steps to prepare text for NLP models.
```
text = "NLP is changing the way machines understand human language!"
text = text.lower()
text = text.translate(str.maketrans("", "", string.punctuation))
tokens = text.split()
stop_words = set(stopwords.words("english"))
tokens = [word for word in tokens if word not in stop_words]
stemmer = PorterStemmer()
tokens = [stemmer.stem(word) for word in tokens]
print(tokens)
```
10. Position in NLP System
Text preprocessing is done before:
- Feature extraction
- Model training
- Prediction or classification
If preprocessing is weak, the entire NLP system becomes weak.
What You Learned in This Chapter
- Meaning of text preprocessing
- Importance of cleaning text
- Steps involved in preprocessing
- Python code for each step
- Difference between stemming and lemmatization
Text preprocessing forms the base of every NLP application. Only after this step can machines start learning from language.
More Nlp tutorials
- Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
- Tokenization in NLP and NLP Project Life Cycle
- Coverting The Text to Vector(one hot encoding and bag of words method)
- Convert text to vector: N-grams and TF-IDF method
- Word Embedding
- What is Natural Language Processing ?
All tutorials · Try the free PySpark compiler · Practice challenges