Embeddings

Nlp tutorial · PySpark.in

Embeddings

Embeddings are dense numerical representations of text (words, sentences, or documents) as vectors in a continuous high-dimensional space, where semantic similarity is captured by geometric proximity. Raw text can't be fed directly into machine learning models it needs to be converted to numbers. Embeddings solve this by mapping words/tokens to vectors such that similar meanings to similar vectors.

For example:

• "king" and "queen" would have vectors close together
• "king" and "banana" would be far apart
• Famous property: king − man + woman ≈ queen

Why Not Just One-Hot Encoding?

One-hot encoding gives each word a sparse binary vector (e.g., 50,000 dimensions for a vocabulary of 50k words). Problems:

• Huge and sparse
• No semantic relationship between words ("cat" and "dog" are equally distant as "cat" and "skyscraper")

Embeddings fix this by being dense (e.g., 300 dimensions) and meaningful.

Types of Embeddings

Word-level (static)

• Word2Vec (Google, 2013) is trained via predicting surrounding words
• GloVe is based on global word co-occurrence statistics
• FastText works on subword units, handles rare/misspelled words

Contextual embeddings (same word, different vector based on context)

• ELMo — uses bidirectional LSTMs
• BERT — transformer-based, hugely influential
• GPT family — unidirectional contextual embeddings

Sentence/Document embeddings

• Sentence-BERT (SBERT) — optimized for semantic similarity tasks
• Universal Sentence Encoder

How Word2Vec Works (Intuition)

Two main training approaches:

• CBOW (Continuous Bag of Words): Predict a word from its context
• Skip-gram: Predict context words from a center word

The model is trained on massive text corpora, and the weight matrices learned during training become the embeddings.

Practical Uses

• Semantic search : find documents by meaning, not keywords
• Recommendation systems : find similar items
• Sentiment analysis : understand tone
• Machine translation
• Clustering & classification
• RAG (Retrieval-Augmented Generation) : powering modern AI systems

Modern Context

In today's LLMs (like GPT-4, Claude), embeddings are generated internally at every layer of the transformer. The input tokens are first converted to embeddings via an embedding lookup table, then refined through attention layers to become rich contextual representations.

More Nlp tutorials

• Natural Language Understanding (NLU) , Natural Language Generation (NLG) and phases of NL
• Tokenization in NLP and NLP Project Life Cycle
• Coverting The Text to Vector(one hot encoding and bag of words method)
• Convert text to vector: N-grams and TF-IDF method
• Word Embedding
• What is Natural Language Processing ?

All tutorials · Try the free PySpark compiler · Practice challenges