Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NLP

Nlp tutorial · PySpark.in

NLP can be broadly divided into two main components:

1. Natural Language Understanding (NLU) – Interpreting meaning and intent from text.

2. Natural Language Generation (NLG) – Producing human-like text from structured or unstructured data.

1. Natural Language Understanding (NLU)

NLU enables machines to understand the meaning, context, and intent behind human language. It powers applications like chatbots, voice assistants, search engines, and content moderation systems.

Key Models and Techniques in NLU

• Transformers:

  • BERT – bidirectional attention for contextual understanding.

  • T5 – treats all tasks as text-to-text.

  • GPT – excels at conversational text generation and comprehension.

• RNNs (LSTM, GRU) – maintain context in sequential text.

• Word Embeddings – Word2Vec, GloVe for semantic similarity.

• Rule-Based Systems – effective for predictable, domain-specific tasks.

• Conditional Random Fields (CRFs) – useful in sequence labeling tasks like POS tagging and NER.

How NLU Works (Step-by-Step)

For the sentence: “A new mobile will be launched in the upcoming year.”

1. Text Preprocessing → tokenization, stopword removal, stemming/lemmatization.

2. POS Tagging → assigns grammatical roles (noun, verb, etc.).

3. Named Entity Recognition (NER) → identifies products, dates, places.

4. Dependency Parsing → analyzes word relationships.

5. Word Sense Disambiguation → resolves multiple meanings (e.g., “mobile” = phone).

6. Intent Recognition → determines intent (e.g., inform about product release).

7. Output Generation → system formulates a response/action.

Applications of NLU

• Virtual assistants (Siri, Alexa, Google Assistant)

• Machine translation

• Search engines (understanding intent)

• Content moderation (hate speech detection)

• Healthcare (interpreting medical notes

2. Natural Language Generation (NLG)

NLG focuses on producing natural-sounding text from structured/semi-structured data. It’s essentially the reverse of NLU.

How does NLG work:

1. Content Determination – selecting relevant data.

2. Document Structuring – organizing content logically.

3. Aggregation – merging facts for fluency.

4. Lexicalization – choosing appropriate words.

5. Referring Expression Generation – using pronouns or entity names.

6. Linguistic Realization – applying grammar rules.

Techniques for Evaluating NLG

• Task-Based Evaluation – does the text help users perform tasks?

• Human Judgment – fluency, coherence, informativeness.

• Automated Metrics – BLEU, ROUGE, METEOR.

Applications of NLG

• Automated reporting (finance, journalism, weather forecasts)

• E-commerce product descriptions

• Chatbots & customer support

• Medical summaries (from Electronic Health Records)

• Business intelligence dashboards (narrating data insights)

Phases of Natural Language Processing

Interpreting meaning and intent from text. It involves a series of phases that work together to process and interpret language with each phase contributing to understanding its structure and meaning. Natural Language Processing (NLP) involves multiple layers of analysis that allow machines to understand and process human language. The major phases include Lexical & Morphological Analysis, Syntactic Analysis, Semantic Analysis, Discourse Integration, and Pragmatic Analysis. 

1. Lexical and Morphological Analysis

•  Lexical Analysis

Lexical analysis focuses on identifying words (lexemes) in a text. It breaks text into tokens (words or phrases) that are meaningful units of language.

Key tasks:

• Tokenization → Splits text into words.
  Example: "I love programming" → ["I", "love", "programming"]

• Part-of-Speech (POS) Tagging → Assigns grammatical categories.
  Example: "I am reading a book" →

  • I → Pronoun (PRP)

  • am → Verb (VBP)

  • reading → Verb (VBG)

  • a → Article (DT)

  • book → Noun (NN)

Importance:

• Breaks text into processable units.

• Simplifies analysis for later NLP stages.

•  Morphological Analysis

Morphological analysis studies morphemes (smallest units of meaning in words).

Key tasks:

• Stemming → reduces words to root form.
  Example: "running" → "run"

• Lemmatization → maps words to base dictionary form.
  Example: "better" → "good"

Importance:

• Helps understand word structure.

• Improves tasks like parsing, translation, and tagging.

2. Syntactic Analysis (Parsing)

Syntactic analysis ensures that words in a sentence follow grammar rules. It creates a parse tree showing subject–verb–object relations.

Key components:

• POS Tagging → assigns grammar roles.

• Ambiguity Resolution → handles words with multiple roles.

Examples:

• Correct: "John eats an apple."

•  Incorrect: "Apple eats John an."

Importance:

• Helps machines understand sentence structure.

• Used in translation, sentiment analysis, and information retrieval.

3. Semantic Analysis

Semantic analysis focuses on meaning of words and sentences. It ensures text is logical, coherent, and contextually valid.

Key tasks:

• Named Entity Recognition (NER): Identifies entities like names, places, dates.
  Example: "Tesla announced a new car in California" → "Tesla" = Organization, "California" = Location.

• Word Sense Disambiguation (WSD): Resolves multiple meanings.
  Example: "bank" → financial institution OR riverbank.

Example:

• "John eats an apple." → semantically valid.

• "Apple eats a John." → grammatically correct but semantically invalid.

Importance:

• Ensures logical interpretation.

• Useful in machine translation, QA systems, and summarization.

4. Discourse Integration

Discourse analysis ensures sentence-to-sentence coherence across larger texts.

Key aspects:

• Anaphora Resolution → resolves pronouns.
  Example: "Taylor went to the store. She bought groceries." → "She" refers to "Taylor".

• Contextual References → clarifies ambiguous expressions.
  Example: "It was a great day." → meaning depends on earlier sentences.

Importance:

• Maintains consistency in long texts.

• Critical for chatbots, translation, and summarization.

5. Pragmatic Analysis

Pragmatic analysis goes beyond literal meaning to capture intent, tone, and context.

Key tasks:

• Understanding Intentions → "Can you pass the salt?" = polite request, not ability check.

• Figurative Meaning → idioms, metaphors.
  Example: "I’m falling for you" = "I love you".

Importance:

• Helps machines detect sarcasm, emotion, and hidden intent.

• Essential for sentiment analysis, conversational AI, and human-like chatbots.

More Nlp tutorials

• Tokenization in NLP and NLP Project Life Cycle
• Coverting The Text to Vector(one hot encoding and bag of words method)
• Convert text to vector: N-grams and TF-IDF method
• Word Embedding
• What is Natural Language Processing ?
• Working with Text in NLP

All tutorials · Try the free PySpark compiler · Practice challenges