​ Understanding Vectors, the Building Blocks of AI

Machine-Learning tutorial · PySpark.in

1. The Problem We Are Solving

We know that machines learn from data. But how does a machine actually "see" a house, a person, or an image? The answer is surprising: everything becomes numbers. But not just any numbers — organized numbers called vectors that have both size and direction.

Given any real-world object, how do we turn it into numbers that a machine can understand and manipulate?

The answer is elegant and comes in two pieces. First, identify the features of the object — the measurable properties. Second, arrange these features as a vector — a list of numbers with direction and magnitude. The total collection is how machines represent the world.

Complete idea: Real Object → Features → Vector → Machine Learning

2. Why This Story Matters

The logic is simple. If you can describe a house by its size, bedrooms, and price, then you can compare houses mathematically. If a machine can represent everything as vectors, then it can calculate distances, similarities, and predictions using simple algebra.

Real-world objects + Feature extraction + Vector math = Machine-readable data

So every system that converts reality into vectors is indeed preparing data for machine learning. The complete picture is exactly the shift from "fuzzy human understanding" to "precise mathematical representation": we take the richness of the world and add structure and computability.

Historical picture: on October 16, 1843, Irish mathematician William Rowan Hamilton had a flash of inspiration while walking along Dublin's Royal Canal. He carved a fundamental formula into the stone of Brougham Bridge: i² = j² = k² = ijk = -1. This discovery of quaternions led to the formalization of vectors — the very tools that would become central to machine learning 180 years later. Mathematicians call this vector analysis; in ML, we call it the language of data.

3. Finding the Vector: From Scalars to Arrows

To understand vectors, we first understand what they are NOT. A scalar is just a single number — like temperature, price, or age. It has magnitude but no direction.

But many things in life need MORE than one number to describe them. Consider a man who walks five miles. If we only know "five miles," we don't know WHERE he ended up. But if we know he walked four miles east and three miles north, we can pinpoint his exact location. This is a vector.

Scalar Example: Temperature

Vector Example: Walking

The vector is an arrow going from (0, 0) to (4, 3), giving both the direction and the distance. The magnitude (length) of the vector is simply the length of the hypotenuse of the right-angled triangle formed by the vector and its components along the x- and y-axes:

|v| = √(4² + 3²) = √(16 + 9) = √25 = 5

So the man walked a total of 5 miles in a straight line from start to finish — even though his journey had two parts (4 east, 3 north).

4. The Complete Solution: Vector Addition

Now imagine the man reaches his destination at (4, 3) and then walks again: two more miles east and six more miles north. Where does he end up?

This is represented by a second vector, an arrow drawn from (4, 3) to (6, 9). This new vector has an x component of 2 and a y component of 6.

The Two Individual Walks:

But what is the NET distance from origin to final destination?

The resultant vector is an arrow drawn from the origin to the final destination (6, 9). Its magnitude is:

|v_total| = √(6² + 9²) = √(36 + 81) = √117 ≈ 10.82 miles

The Magic of Vector Addition:

Instead of calculating the long path, we can simply ADD the components:

v1 + v2 = (4 + 2, 3 + 6) = (6, 9)

This is the resultant vector! The net distance in the xy coordinate space is 10.82 miles — less than the total walked because the walks partly overlapped in direction.

This now helps us make sense of what Newton was saying. Let's say the acceleration caused by one force acting upon an object is given by the vector (2, 6) and that the acceleration caused by another force on the same object is given by the vector (4, 3). Both forces are acting on the object at the same time. What is the total acceleration of the object? According to Newton's corollary, the geometric interpretation involves drawing a parallelogram — the net acceleration is given by the diagonal vector (6, 9).

5. Vectors by the Numbers: No Arrows Needed!

Vector analysis doesn't have to be geometric. It can come down to manipulating numbers written in a certain format. And in fact, for machine learning, that's how we need to think about vectors.

For example, the accelerations caused by the two forces in the previous example are simply arrays of two numbers each: [4, 3] and [2, 6]. Adding them is the same as adding the individual components of each vector:

Subtracting vectors is similar:

What just happened? Why is the y component of the resultant vector negative? If these numbers still represent acceleration, then subtraction meant that the second force was acting against the first force; along the x-axis, the acceleration is just a little bit less than when we were adding the two vectors, but it is still positive; along the y-axis, however, the force is now acting against the initial direction of motion, resulting in a deceleration.

Multiplying a Vector by a Scalar:

One can multiply a vector by a scalar — simply multiply each element of the vector by the scalar:

Geometrically, that's the same as stretching the arrow five times in the same direction. The magnitude of the original vector is 5. Scaling it 5 times gives us a new magnitude of 25. If you were to calculate the magnitude of the new vector using its scaled-up coordinates, you'd again get:

√(20² + 15²) = √(400 + 225) = √625 = 25

There's yet another way to represent vectors. Restricting ourselves to two dimensions, think of a vector of length one, i, along the x-axis and a vector of length one, j, along the y-axis. Note that i and j are in lowercase and boldface; this signifies that they are vectors. So, i can be thought of as an arrow that points from (0, 0) to (1, 0) and j as an arrow that points from (0, 0) to (0, 1). Each has a magnitude of 1 and is also called a unit vector.

Given this, the vectors (4, 3) and (2, 6), in Cartesian coordinates, can be written as 4i + 3j and 2i + 6j, respectively. That's the same as saying that the vector (4, 3) is 4 units along the x-axis and 3 units along the y-axis and that the vector (2, 6) is 2 units along the x-axis and 6 units along the y-axis. The use of i and j is shorthand for representing vectors.

It's also important to point out that a unit vector is simply a vector with a magnitude of 1; it doesn't have to lie along the perpendicular axes of some coordinate space.

These ideas apply to higher dimensions, too, and we'll come to that. For now, getting a handle on the mathematical manipulation of 2D vectors and their corresponding geometric meanings will go a long way toward helping us understand the role of their higher-dimensional counterparts in machine learning.

6. The Dot Product: Where Geometry Meets Algebra

Another important operation with vectors is something called the dot product. Consider vector (4, 0), call it a, and vector (5, 5), call it b. (Again, the boldface and lowercase for letters a and b signify that they are vectors.)

Conceptually, the dot product a · b — read that as "a dot b" — is defined as the magnitude of a multiplied by the projection of b onto a, where the projection can be thought of as the "shadow cast" by one vector onto another.

Geometric View:

a · b = |a| × |b| × cos(θ)

Where θ is the angle between the two vectors. This equals (length of a) × (shadow of b on a).

Algebraic View:

Say, a = a1i + a2j and b = b1i + b2j. Then:

a · b = (a1i + a2j) · (b1i + b2j)

= a1b1 × i·i + a1b2 × i·j + a2b1 × j·i + a2b2 × j·j

Note that the second and third terms in the equation turn out to be zero. The vectors i and j are orthogonal (perpendicular), so i·j and j·i are zero. Also, both i·i and j·j equal 1. All we are left with is a scalar quantity:

a · b = a1b1 + a2b2

Example:

What Does Dot Product Tell Us?

Table

Situation

Angle

Dot Product

Meaning

Same direction

0°

Positive (maximum)

Vectors agree completely

Perpendicular

90°

Zero!

Vectors are independent

Opposite direction

180°

Negative (minimum)

Vectors disagree completely

Dot product measures HOW MUCH two vectors POINT THE SAME WAY

7. The Four Cases: Vectors in Machine Learning

Given different types of data, there are four important ways vectors are used in ML.

Case 1: Feature Vectors — Representing Objects

Every object becomes a vector of features.

Example: House = [3 bedrooms, 1500 sq ft, 2 bathrooms, 1 garage]

This is a 4-dimensional vector: v = (3, 1500, 2, 1)

Case 2: Weight Vectors — Making Predictions

The perceptron's weights form a vector. The prediction is the dot product of weights and inputs!

Example: weights = (0.5, 0.001, 0.3, 0.2), inputs = (3, 1500, 2, 1)

Prediction = 0.5×3 + 0.001×1500 + 0.3×2 + 0.2×1 = weighted sum = dot product!

Case 3: Similarity Vectors — Finding Matches

Dot product tells us how similar two things are.

Example: Two movies both liked by the same people → their vectors have high dot product → recommend one to fans of the other!

Case 4: Distance Vectors — Measuring Difference

The distance between two vectors tells us how different two data points are.

Example: A new data point far from all known examples might be an outlier or anomaly.

Summary: The dimension of the vector tells you how many features describe the object. The dot product tells you how similar two objects are. Vector addition combines effects. Vector scaling amplifies or reduces. This four-case analysis is the master key to understanding why vectors are the language of machine learning.

Python Implementation: Vectors from Scratch

Below is a complete, zero-library implementation of vectors, vector addition, scalar multiplication, and dot product. This is exactly how the math works inside every ML algorithm.

```

python

# ============================================

# BLOG 4: VECTORS, VECTOR ADDITION, DOT PRODUCT

# Zero-library implementation

# ============================================

import math

class Vector:

"""

A simple vector class for machine learning.

No numpy needed - just pure Python!

"""

def __init__(self, components):

"""Create a vector from a list of numbers"""

self.components = list(components)

self.dimension = len(components)


def __repr__(self):

return f"Vector({self.components})"


def __str__(self):

return str(self.components)


def magnitude(self):

"""Calculate the length (size) of the vector"""

# |v| = sqrt(v1² + v2² + ... + vn²)

total = 0

for c in self.components:

total += c ** 2

return math.sqrt(total)


def __add__(self, other):

"""Add two vectors: just add each component"""

if self.dimension != other.dimension:

raise ValueError("Vectors must have same dimension!")


result = []

for i in range(self.dimension):

result.append(self.components[i] + other.components[i])

return Vector(result)


def __sub__(self, other):

"""Subtract two vectors: subtract each component"""

if self.dimension != other.dimension:

raise ValueError("Vectors must have same dimension!")


result = []

for i in range(self.dimension):

result.append(self.components[i] - other.components[i])

return Vector(result)


def scale(self, scalar):

"""Multiply vector by a number (scalar multiplication)"""

result = []

for c in self.components:

result.append(c * scalar)

return Vector(result)


def dot_product(self, other):

"""Calculate dot product: multiply components and sum"""

if self.dimension != other.dimension:

raise ValueError("Vectors must have same dimension!")


result = 0

for i in range(self.dimension):

result += self.components[i] * other.components[i]

return result


def angle_between(self, other):

"""Calculate angle between two vectors (in degrees)"""

dot = self.dot_product(other)

mag_a = self.magnitude()

mag_b = other.magnitude()


if mag_a == 0 or mag_b == 0:

return 0


# cos(θ) = (a·b) / (|a| × |b|)

cos_theta = dot / (mag_a * mag_b)

# Clamp to [-1, 1] to avoid floating point errors

cos_theta = max(-1, min(1, cos_theta))

theta_rad = math.acos(cos_theta)

theta_deg = math.degrees(theta_rad)

return theta_deg

# ============================================

# DEMONSTRATION: ALL VECTOR OPERATIONS

# ============================================

print("=" * 60)

print("VECTORS: THE BUILDING BLOCKS OF MACHINE LEARNING")

print("=" * 60)

# Create vectors from the book's walking example

v1 = Vector([4, 3]) # Walk 1: 4 miles east, 3 miles north

v2 = Vector([2, 6]) # Walk 2: 2 miles east, 6 miles north

print(f"\nVector v1 (first walk): {v1}")

print(f"Vector v2 (second walk): {v2}")

print(f"\n{'='*60}")

print("1. MAGNITUDE (Length of the vector)")

print(f"{'='*60}")

print(f"|v1| = √(4² + 3²) = √{4**2 + 3**2} = {v1.magnitude()}")

print(f"|v2| = √(2² + 6²) = √{2**2 + 6**2} = {v2.magnitude():.2f}")

print(f"\n{'='*60}")

print("2. VECTOR ADDITION (v1 + v2)")

print(f"{'='*60}")

v_total = v1 + v2

print(f"v1 + v2 = ({v1.components[0]} + {v2.components[0]}, {v1.components[1]} + {v2.components[1]})")

print(f" = ({v_total.components[0]}, {v_total.components[1]})")

print(f"|v_total| = √({v_total.components[0]}² + {v_total.components[1]}²) = {v_total.magnitude():.2f}")

print(f"Total distance walked: {v1.magnitude() + v2.magnitude():.2f} miles")

print(f"Net displacement: {v_total.magnitude():.2f} miles")

print(f"\n{'='*60}")

print("3. VECTOR SUBTRACTION (v1 - v2)")

print(f"{'='*60}")

v_diff = v1 - v2

print(f"v1 - v2 = ({v1.components[0]} - {v2.components[0]}, {v1.components[1]} - {v2.components[1]})")

print(f" = ({v_diff.components[0]}, {v_diff.components[1]})")

print(f"\n{'='*60}")

print("4. SCALAR MULTIPLICATION (5 × v1)")

print(f"{'='*60}")

v_scaled = v1.scale(5)

print(f"5 × v1 = (5×{v1.components[0]}, 5×{v1.components[1]})")

print(f" = ({v_scaled.components[0]}, {v_scaled.components[1]})")

print(f"|5×v1| = {v_scaled.magnitude():.1f} (should be 5 × {v1.magnitude()} = {5 * v1.magnitude()})")

print(f"\n{'='*60}")

print("5. DOT PRODUCT (v1 · v2)")

print(f"{'='*60}")

dot = v1.dot_product(v2)

print(f"v1 · v2 = {v1.components[0]}×{v2.components[0]} + {v1.components[1]}×{v2.components[1]}")

print(f" = {v1.components[0] * v2.components[0]} + {v1.components[1] * v2.components[1]}")

print(f" = {dot}")

# Verify with geometric formula

cos_theta = dot / (v1.magnitude() * v2.magnitude())

print(f"\nGeometric check:")

print(f"cos(θ) = (v1·v2) / (|v1|×|v2|) = {dot} / ({v1.magnitude():.2f} × {v2.magnitude():.2f}) = {cos_theta:.4f}")

angle = v1.angle_between(v2)

print(f"Angle between v1 and v2: {angle:.1f} degrees")

print(f"\n{'='*60}")

print("6. DOT PRODUCT INTERPRETATION")

print(f"{'='*60}")

# Create test vectors to show dot product meaning

same_dir = Vector([1, 0])

perp = Vector([0, 1])

opposite = Vector([-1, 0])

print(f"Same direction: (1,0)·(1,0) = {Vector([1,0]).dot_product(Vector([1,0]))} (positive = agree)")

print(f"Perpendicular: (1,0)·(0,1) = {Vector([1,0]).dot_product(Vector([0,1]))} (zero = independent)")

print(f"Opposite direction: (1,0)·(-1,0) = {Vector([1,0]).dot_product(Vector([-1,0]))} (negative = disagree)")

print(f"\n{'='*60}")

print("7. MACHINE LEARNING EXAMPLE: House Price Prediction")

print(f"{'='*60}")

# Represent a house as a vector

house = Vector([3, 1500, 2, 1]) # [bedrooms, sqft, bathrooms, garage]

weights = Vector([50, 0.1, 30, 20]) # weights for each feature

print(f"House features: {house}")

print(f"Weights: {weights}")

print(f"Predicted price = house · weights")

price = house.dot_product(weights)

print(f" = 3×50 + 1500×0.1 + 2×30 + 1×20")

print(f" = {3*50} + {1500*0.1} + {2*30} + {1*20}")

print(f" = {price} thousand dollars")

print(f" = ${price*1000:,}")

print(f"\n{'='*60}")

print("8. SIMILARITY EXAMPLE: Movie Recommendations")

print(f"{'='*60}")

# Two users' movie ratings (1-5 scale)

user_alice = Vector([5, 4, 1, 2]) # [Action, Comedy, Horror, Romance]

user_bob = Vector([4, 5, 1, 1]) # [Action, Comedy, Horror, Romance]

similarity = user_alice.dot_product(user_bob)

print(f"Alice's ratings: {user_alice}")

print(f"Bob's ratings: {user_bob}")

print(f"Similarity (dot product): {similarity}")

print(f"High dot product = similar tastes = recommend same movies!")

print(f"\n{'='*60}")

print("SUMMARY: WHAT DID WE LEARN?")

print(f"{'='*60}")

print("""

✅ Vectors = lists of numbers with direction and size

✅ Vector addition = add components one by one

✅ Scalar multiplication = stretch or shrink the vector

✅ Dot product = measure of how much vectors agree

✅ In ML: EVERYTHING is a vector (houses, users, images, words)

✅ Prediction = dot product of features and weights

✅ Similarity = dot product of two user/item vectors

This is the mathematical foundation of ALL machine learning!

""")

```

This is the mathematical foundation of ALL machine learning!

The Four Cases: Vector Dimensions in ML

Case

Vector Type

Dimension

Example Use

1

2D Vector

2

Simple graphs, walking directions

2

Feature Vector

5-50

House features, customer profiles

3

Image Vector

400-1M

Pixels of a photo (20×20 = 400)

4

Word/Document Vector

100-10K

Text embeddings, semantic meaning

Complete solution = real object → feature list → vector → vector math (add, scale, dot) → prediction. The dimension comes from the number of features. The power comes from dot products and geometric relationships.

Why Vectors Were Revolutionary

None of this was formally understood at the beginnings of vector analysis when Newton published his Principia in 1687. His contemporary Gottfried Wilhelm Leibniz (1646–1716), however, had more than an inkling about this new way of thinking. In 1679, in a letter to another luminous contemporary, Christiaan Huygens, Leibniz wrote, "I believe that I have found the way . . . that we can represent figures and even machines and movements by characters, as algebra represents numbers or magnitudes." Leibniz never quite formalized his intuition, but his prescience — as we'll see when we understand the importance of vectors for machine learning — was astounding.

Following Leibniz, a host of other mathematicians, including Johann Carl Friedrich Gauss (1777–1855), developed methods for the geometric representation of certain types of numbers in two dimensions, setting the stage for Hamilton's discovery of quaternions and the formalization of vector analysis.

Today, vectors are not just abstract math — they are the language of data. Every machine learning system converts the world into vectors, manipulates them with dot products and additions, and converts the results back into useful predictions. Without Hamilton's flash of inspiration on Brougham Bridge, modern AI would not exist.

Try It Yourself!

Exercise 1: Calculate the magnitude of vector v = (6, 8). What famous triangle does this remind you of?

Exercise 2: If a = (3, 4) and b = (1, 2), find a + b, a - b, and a · b.

Exercise 3: Two vectors have dot product = 0. What does this tell you about their angle?

Exercise 4: A house has features [4, 2000, 3, 1] and weights are [40, 0.08, 25, 15]. What's the predicted price?

References & Further Reading

Anil Ananthaswamy. Why Machines Learn: The Elegant Math Behind Modern AI. Dutton, 2024.

William Rowan Hamilton. Letter to his son, October 16, 1843. (The original quaternion discovery on Brougham Bridge)

3Blue1Brown. Essence of Linear Algebra series. YouTube. (Chapters on vectors and dot products)

Khan Academy. Vectors and spaces. khanacademy.org/math/linear-algebra

More Machine-Learning tutorials

All tutorials · Try the free PySpark compiler · Practice challenges