Naive Bayes for Text Classification

Nlp tutorial · PySpark.in

Naive Bayes for Text Classification in Natural Language Processing (NLP)

Naive Bayes is one of the most fundamental and widely used supervised machine learning algorithms in the field of Natural Language Processing (NLP). It is primarily employed for solving classification problems such as spam detection, sentiment analysis, document categorization, and email filtering. The algorithm is based on the principles of probability theory and Bayes’ Theorem. Despite its simplicity, Naive Bayes performs remarkably well for textual data classification tasks

Introduction to Naive Bayes

Naive Bayes is a probabilistic classifier that predicts the category of a given text document by calculating the probability of the document belonging to a particular class. The term “Naive” refers to the assumption that all features are independent of one another. For example, in a sentence such as:

the algorithm assumes that the words “movie”, “very”, and “good” contribute independently toward the classification result. Although this assumption is not entirely realistic, the algorithm still provides highly effective results in many real-world NLP applications.

Bayes’ Theorem

Naive Bayes is derived from Bayes’ Theorem, which describes the probability of an event based on prior knowledge of conditions related to the event. In text classification, the algorithm computes the probability of a document belonging to each class and assigns the document to the class with the highest probability.

The mathematical representation is:

Where:

P(C∣X)P(C|X)P(C∣X) = Posterior Probability

P(X∣C)P(X|C)P(X∣C) = Likelihood Probability

P(C)P(C)P(C) = Prior Probability

P(X)P(X)P(X) = Evidence Probability

Why Naive Bayes is Suitable for NLP

Natural Language Processing tasks involve handling large textual datasets containing thousands of words. Naive Bayes is highly suitable for NLP because:

It performs efficiently on high-dimensional data.

It requires less training data.

It is computationally fast.

It provides good accuracy for text classification tasks.

It works well with sparse datasets

Workflow of Naive Bayes in NLP

The implementation of Naive Bayes in NLP generally follows the following stages.

Step 1: Collection of Text Data

The first step involves gathering textual data for training the model.

Example Dataset

Text	Category
“Win money now”	Spam
“Meeting at 5 pm”	Ham
“Claim your prize”	Spam
“Project discussion tomorrow”	Ham

The dataset contains text samples and their corresponding labels

Step 2: Text Preprocessing

Raw textual data contains unnecessary symbols and inconsistencies. Therefore, preprocessing is performed to clean the data.

Common Preprocessing Techniques

Lowercasing

Removal of punctuation

Tokenization

Stop-word removal

Stemming

Lemmatization

Example

Before preprocessing:

After preprocessing:

Step 3: Feature Extraction

Machine learning algorithms cannot directly process textual data. Therefore, text is converted into numerical vectors.

Two commonly used techniques are:

Bag of Words (BoW)

TF-IDF (Term Frequency–Inverse Document Frequency)

Example Vocabulary

Sentence:

Vector Representation:

This numerical representation becomes the input for the classifier.

Step 4: Training the Naive Bayes Model

During training, the algorithm calculates the probabilities associated with each class.

For spam detection:

The model predicts the class that has the maximum posterior probability.

Laplace Smoothing

One major problem in Naive Bayes is the occurrence of zero probability. If a word does not appear in the training data for a particular class, its probability becomes zero, which may affect the entire prediction.

To overcome this issue, Laplace Smoothing is applied.

The formula is:

This technique is also known as Add-One Smoothing.

Applications of Naive Bayes in NLP

Naive Bayes has several practical applications in Natural Language Processing.

Application	Example
Spam Detection	Gmail spam filtering
Sentiment Analysis	Positive/negative review classification
News Categorization	Sports, politics, technology
Language Detection	English, Hindi, French
Email Classification	Promotions, updates, social

Practical Implementation Using Python

Problem Statement

Develop a spam email classifier using Multinomial Naive Bayes.

Installation of Required Libraries

```

pip install pandas scikit-learn

```

Example:

```

import pandas as pd

from sklearn.feature_extraction.text import CountVectorizer from sklearn.model_selection import train_test_split from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import accuracy_score, classification_report

Creating dataset

data = {

"text": [

    "Win money now",

    "Claim your free prize",

    "Meeting at 10 am",

    "Project discussion tomorrow",

    "Congratulations you won lottery",

    "Let us complete assignment"

],

"label": [

    "spam",

    "spam",

    "ham",

    "ham",

    "spam",

    "ham"

]

}

Creating DataFrame

df = pd.DataFrame(data)

Input and output variables

X = df["text"]

y = df["label"]

Converting text into vectors

vectorizer = CountVectorizer()

X_vectorized = vectorizer.fit_transform(X)

Splitting dataset

X_train, X_test, y_train, y_test = train_test_split(

X_vectorized,

y,

test_size=0.3,

random_state=42

)

Creating Naive Bayes model

model = MultinomialNB()

Training model

model.fit(X_train, y_train)

Prediction

y_pred = model.predict(X_test)

Accuracy

print("Accuracy:", accuracy_score(y_test, y_pred))

Classification report

print(classification_report(y_test, y_pred))

Testing custom message

message = ["Free lottery prize available"]

message_vector = vectorizer.transform(message)

prediction = model.predict(message_vector)

print("Prediction:", prediction[0])

```

Conclusion

Naive Bayes is one of the most efficient and beginner-friendly algorithms for text classification in Natural Language Processing. Its simplicity, fast computation, and strong performance on textual datasets make it an ideal choice for spam filtering, sentiment analysis, and document categorization tasks. Although the independence assumption is simplistic, the algorithm continues to remain highly effective in practical NLP applications.

Naive Bayes for Text Classification

Introduction to Naive Bayes

More Nlp tutorials