Understanding Vector Spaces

Linear-Algebra tutorial · PySpark.in

1.1 What This Chapter is About

In Chapter 1, we learned that a vector has a magnitude (length) and a direction, and that vectors live inside a structure called a vector space. In this chapter, we go one level deeper and ask two fundamental questions:

First: What are the simplest, most "atomic" vectors from which every other vector can be built? (This leads us to basis vectors.)

Second: If we combine a set of vectors in every possible way, how much of space can they cover? (This leads us to span.)

Understanding these two ideas — basis and span — is the heart of linear algebra. They explain why transformations, machine learning models, and coordinate systems work the way they do.

1.2 Basis Vectors

Think about how a number line works. To reach the number 5, you simply start at zero and scale the unit "1" five times. The number 1 is the fundamental building block — every other number on the line is just a scaled version of it.

Basis vectors work the same way, but in 2D (or 3D) space. They are the smallest, most fundamental arrows from which you can build any other vector in that space, just by scaling and combining them.

Basis Vector: A basis vector is a unit-length vector that points along one of the fundamental axes of a coordinate system. It is the "building block" of its dimension.

Basis Vectors in 2D Space

In the 2D coordinate plane, we use two basis vectors. î (called "i-hat") points one unit along the x-axis — purely east. ĵ (called "j-hat") points one unit along the y-axis — purely north.

î = (1, 0) ĵ = (0, 1)

These two directions, horizontal and vertical, are completely independent of each other — moving along î has absolutely no effect on the ĵ direction, and vice versa. Because of this independence, they together give us complete control over the entire 2D plane.

Think of it like a city grid. To get anywhere in a city, you only need two types of movement: East-West (î) and North-South (ĵ). No matter where your destination is, those two directions are always enough.

Why Can î and ĵ Reach Every Single Point in 2D Space?

This is the key question. Here is the careful reasoning. Any point in 2D space is defined by two coordinates — an x-value and a y-value. To reach the point (a, b), all you need to do is:

Scale î by a: Move "a" units along the x-axis. This covers the entire horizontal component of your journey.
Scale ĵ by b: Move "b" units along the y-axis. This covers the entire vertical component.
Add the two results: The combined arrow lands exactly at (a, b).

Mathematically, we write this as:

v⃗ = a·î + b·ĵ = a·(1, 0) + b·(0, 1) = (a, b)

Since every possible point in 2D has an x-value and a y-value, and since î covers x and ĵ covers y, there is no point that this combination cannot reach. This is why {î, ĵ} is called a basis of ℝ² (2D real space).

Figure 1.1 — To represent v⃗ = (2, 3): scale î by 2 (move right), then scale ĵ by 3 (move up). The combination lands exactly at (2, 3).

Geometric intuition: Any movement in 2D can always be split into a purely horizontal part and a purely vertical part. That is all basis vectors are — they represent those two pure, independent directions.

But Why Not Just Use Coordinates Directly?

You might wonder: if I already know a point is at (4, 3), why bother writing it as 4î + 3ĵ? It seems like extra work.

The answer is that the coordinate system you use is a choice, not a law of nature. When we write (4, 3), we are silently assuming the standard grid with î and ĵ as the rulers. But in linear algebra, we frequently need to change that grid — rotating it, stretching it, or shearing it. When you change the basis vectors, all the coordinates in the space change with them.

Thinking explicitly in terms of î and ĵ makes that transformation process visible and understandable. Basis vectors are the lens through which transformations are understood in linear algebra.

1.3 Basis Vectors in 3D Space

The same idea scales up to three dimensions naturally. In 3D space, we need three basis vectors — one for each independent direction:

î = (1, 0, 0) — points one unit along the x-axis (east)
ĵ = (0, 1, 0) — points one unit along the y-axis (north)
k̂ = (0, 0, 1) — points one unit along the z-axis (up)

Any vector in 3D space can be written as a combination of these three:

v⃗ = a·î + b·ĵ + c·k̂ = (a, b, c)

To reach any point in 3D, you move "a" units east, then "b" units north, then "c" units up. The three directions are completely independent — moving in any one of them has zero effect on the others.

Real-world example: imagine you are inside a building. To describe any location, you need three independent instructions: "go this many metres forward (x), this many metres sideways (y), and take the lift to this floor (z)." That is exactly î, ĵ, and k̂ at work.

Just as {î, ĵ} forms a basis for ℝ², {î, ĵ, k̂} forms a basis for ℝ³ — the entire 3D space. Every point, every direction, every vector in 3D can be reached by a combination of these three building blocks.

1.4 Span — How Much Space Can Vectors Cover?

Now that we understand basis vectors, we can ask a more general question: if I give you any two vectors — not necessarily î and ĵ — and let you scale and combine them in every possible way, what is the set of all vectors you can produce?

That set is called the span of those vectors.

Span: The span of a set of vectors is the collection of ALL possible vectors that can be formed by taking linear combinations (scaling and adding) of those vectors.

A linear combination of two vectors v⃗ and w⃗ looks like this, where a and b can be any real numbers:

a·v⃗ + b·w⃗

By changing a and b freely, we "sweep out" a region of space. The shape of that region depends critically on whether v⃗ and w⃗ point in genuinely different directions.

Figure 1.2 — When v⃗ and w⃗ point in different (linearly independent) directions, their linear combinations sweep out the entire 2D plane.

Intuition: Fixing One Vector and Varying the Other

Here is a powerful way to build intuition for span. Fix the scalar a so that a·v⃗ is a single, stationary arrow. Now let b vary freely.

As b changes, b·w⃗ stretches and shrinks — always pointing in the direction of w⃗, but with a changing length. When you add this sliding arrow to the fixed arrow a·v⃗, the tip of the result traces out a straight line through the plane.

Now also let a vary. Each different value of a creates a different fixed arrow, and each produces a different straight line. As a sweeps through all values, these parallel lines stack up and fill the entire plane — provided v⃗ and w⃗ point in different directions.

Think of it like painting a wall. One brush stroke (fixing a, varying b) covers a single line. Stacking infinitely many parallel brush strokes (also varying a) covers the entire wall.

1.5 Linear Independence and Dependence

The phrase "genuinely different directions" is doing a lot of work above. Let us make it precise by looking at two distinct cases.

Case 1: Vectors in Different Directions (Linearly Independent)

If v⃗ and w⃗ point in genuinely different directions — meaning neither one is just a stretched or flipped version of the other — then varying both scalars a and b gives you access to every point in the 2D plane. Their span is the whole plane, ℝ².

These vectors are called linearly independent. The word "independent" is fitting: each vector contributes something genuinely new. v⃗ introduces a direction that w⃗ cannot replicate, and vice versa.

Linearly Independent: Two vectors are linearly independent if neither can be written as a scalar multiple of the other. Each one introduces a brand-new direction to the space.

Case 2: Vectors Pointing in the Same Direction (Linearly Dependent)

Now suppose w⃗ = k·v⃗ for some number k — meaning w⃗ is just v⃗ scaled up, or v⃗ scaled and flipped. Both vectors lie on the same line through the origin.

What happens to our linear combination?

a·v⃗ + b·w⃗ = a·v⃗ + b·(k·v⃗) = (a + b·k)·v⃗

No matter what values a and b take, the result is always a scalar multiple of v⃗. The two scalars a and b collapse into a single number (a + b·k). You have effectively only one degree of freedom, not two.

This means all reachable vectors lie on a single straight line — the line in the direction of v⃗. You can never "step sideways" into the rest of the plane.

Linearly Dependent: Two vectors are linearly dependent if one is a scalar multiple of the other (they lie on the same line). Their linear combinations can only span a line, not a plane.

Memorable analogy: imagine two people trying to row a boat. If both oars point in exactly the same direction, they can only move the boat forward or backward — never sideways. You need two oars pointing in different directions to steer anywhere on the water. That is linear independence.

Why Does This Matter?

Linear independence is not just a mathematical curiosity — it has direct, practical consequences:

Span: Only linearly independent vectors can span a full 2D plane. Dependent vectors are "redundant" — one of them adds no new information.
Basis: For a set of vectors to form a basis, they must be linearly independent. Otherwise the basis is broken — it cannot reach all of the space.
Data and Machine Learning: In a dataset, linearly dependent features are redundant — one feature is just a scaled copy of another, and including both wastes resources and can cause problems in models.

Chapter Summary

Here is a concise review of every key idea from this chapter, in the order they build on each other.

Basis vectors are the fundamental unit-length building blocks of a coordinate space. In 2D, they are î and ĵ. In 3D, they are î, ĵ, and k̂.
Any vector in 2D can be written as v⃗ = a·î + b·ĵ, because î controls the horizontal direction and ĵ controls the vertical direction — independently.
In 3D, any vector can be written as v⃗ = a·î + b·ĵ + c·k̂, with the third basis vector k̂ controlling depth.
Span is the set of all vectors reachable by linear combinations (a·v⃗ + b·w⃗) of a given set of vectors.
Linearly independent vectors point in genuinely different directions. Their span fills an entire plane (or higher-dimensional space).
Linearly dependent vectors lie on the same line (one is a multiple of the other). Their span is only a line — they cannot fill a plane.

References

Ananthaswamy, Anil. Why Machines Learn. Dutton, 2018.

3Blue1Brown. "Essence of Linear Algebra" series. YouTube, 2016.

Wikipedia — Vector Space. https://en.wikipedia.org/wiki/Vector_space

End of Dot Product Blog — Linear Algebra Series