N-gram Language Models

Nlp tutorial · PySpark.in

N-gram Language Models

A Language Model (LM) is a probabilistic model that assigns a probability to a sequence of words. In other words, it estimates how likely a particular sequence of words is to occur in a language. Language models are widely used to predict the next word in a sentence based on the words that precede it. For instance, consider the following sentence:

all of a sudden I notice three guys standing on the sidewalk

This sentence is grammatically and semantically correct. However, if the same words are rearranged arbitrarily:

one guys all I notice sidewalk three a sudden standing the

the resulting sequence becomes incoherent. This demonstrates that language models capture not only the presence of words but also their correct order and contextual relationships.

N-gram Language Models

One of the simplest forms of language models is the N-gram model. An N-gram is defined as a contiguous sequence of N words in a given text.

The N-gram model estimates the probability of a word based on the previous (N−1) words. Thus, it simplifies the complex problem of predicting word sequences by considering only a limited context.

N-Grams

computing P(w|h), the probability of a word w given some history h. Suppose the history h is “The water of Walden Pond is so beautifully ” and we want to know the probability that the next word is blue:

One way to estimate this probability is directly from relative frequency counts: take a very large corpus, count the number of times we see The water of Walden Pond is so beautifully, and count the number of times this is followed by blue. This would be answering the question “Out of the times we saw the history h, how many times was it followed by the word w”, as follows:

Intuition of Chain Rule in Language Models

We want to compute the probability of a sequence of words:

This means:What is the probability that this whole sentence occurs?

Problem

Computing this directly is very difficult Because:

Solution is simple use Chain Rule of Probability

We break the big problem into smaller parts using:

EXAMPLE: Applying to Words

For a sentence:

“I love NLP”

We compute:

Expanded form:

Example

Sentence:
“I love NLP”

BUT Problem still exists

Even this is complex because:

depends on all previous words

Final simplification (N-gram INTUTION)

We assume: Only last few words matter

Example:

The Markov assumption

The bigram model, for example, approximates the probability of a word given all the previous words P(wn|w1:n−1) by using only the conditional probability given the preceding word P(wn|wn−1). In other words, The assumption that the probability of a word depends only on the previous word is called a Markov assumption. Markov models are the class of probabilistic models that assume we can predict the probability of some future unit without looking too far into the past

When we use a bigram model to predict the conditional probability of the next word, we are thus making the following approximation:-🡪

How to estimate probabilities

An intuitive way to estimate probabilities is called maximum likelihood estimation or MLE. We get maximum likelihood for the parameters of an n-gram model by getting counts from normalize a corpus, and normalizing the counts so that they lie between 0 and 1. bigram probability of a word wn given a previous word wn−1, we’ll compute the count of the bigram

Let’s work through an example using a mini-corpus of three sentences. We’ll first need to augment each sentence with a special symbol </s>

For the general case of MLE n-gram parameter estimation:

EXAMPLE:

fig.1

fig.2

Fig 2 shows bigram probabilities after normalization (dividing each cell in Fig. (1) by the appropriate unigram for its row, taken from the following set of unigram counts)

Here are a few other useful probabilities:

Now we can compute the probability of sentences like I want English food or I want Chinese food by simply multiplying the appropriate bigram probabilities together, as follows:

Dealing with scale in large n-gram models: Log probabilities Language model probabilities are always stored and computed in log space aslog probabilities. This is because probabilities are (by definition) less than or equal to 1, and so the more probabilities we multiply together, the smaller the product becomes. Adding in log space is equivalent to multiplying in linear space, so we combine log probabilities by adding them

Example Problem Suppose we have 4 probabilities:

Convert to Log Space Take natural log(log)

Add Logs

Convert Back

More Nlp tutorials

All tutorials · Try the free PySpark compiler · Practice challenges