Spaces of Vectors

Linear-Algebra tutorial · PySpark.in

A Beginner's Guide to Linear Algebra | Part 4 of 4

Spaces of Vectors

1. What Is a Vector Space?

You already know what a vector looks like — a column of numbers. But now we want to think bigger. What if we collected all possible vectors of a certain kind into a single mathematical object? That collection, if it behaves in a specific way, is called a vector space.

A vector space is a set of vectors that is closed under the two operations you already know: addition and scalar multiplication. 'Closed' simply means: if you add two vectors from the set, the result is still in the set. If you scale a vector from the set, the result is still in the set. Nothing ever escapes.

A vector space is any collection of vectors that is closed under addition and scalar multiplication. The two most important examples are Rⁿ (all n-dimensional column vectors) and the zero space {0}.

The most familiar vector space is Rⁿ — the set of all column vectors with n real number entries. R² is the full plane. R³ is all of three-dimensional space. These sets are closed: add two vectors in R² and you get a vector in R², multiply by any scalar and you stay in R². They are vector spaces.

2. Subspaces — Spaces Inside Spaces

Here is where it gets interesting. Inside a large vector space, you can often find smaller collections that are themselves vector spaces. These are called subspaces.

A subspace of a vector space V is any subset S of V that is also a vector space in its own right. For S to qualify, it must pass three simple tests:

1. The zero vector must be in S.

2. If u and v are in S, then u + v must also be in S.

3. If v is in S and c is any scalar, then cv must also be in S.

The third test automatically implies the first (set c = 0). And together, tests 2 and 3 say that all linear combinations of vectors in S must stay in S.

Think of a subspace as a flat object that passes through the origin. In R³, the valid subspaces are: the origin alone, any line through the origin, any plane through the origin, and all of R³ itself. A line or plane that does not pass through the origin is NOT a subspace — it fails the zero vector test.

This 'through the origin' requirement is crucial and comes up constantly. A tilted plane in R³ that does not contain (0,0,0) is not a subspace, even though it looks like a flat 2D object. Add the zero vector to it and it is no longer in the set.

3. The Column Space C(A)

Now we connect subspaces directly to matrices. Given any matrix A, we can form a very natural subspace: take all possible linear combinations of the columns of A. This collection is called the column space of A, written C(A).

If A has columns c1, c2, ..., cn then:

C(A) = { x1·c1 + x2·c2 + ... + xn·cn for all choices of x1, x2, ..., xn }

In other words, C(A) consists of every vector that can be produced by multiplying A on the right by some vector x. Said differently: C(A) is exactly the set of all vectors b for which Ax = b has at least one solution.

The column space C(A) is a subspace of Rᵐ (where m is the number of rows of A). The equation Ax = b has a solution if and only if b is in C(A).

Let's look at a concrete example. Suppose:

A = [ 1 0 ]

[ 0 1 ]

[ 0 0 ]

The columns of A are (1,0,0) and (0,1,0). Their linear combinations are all vectors of the form (x, y, 0) — the entire xy-plane inside R³. Any vector b with a nonzero third component cannot be reached. C(A) is the xy-plane, a 2D subspace of R³.

4. The Row Space

Just as we formed the column space from the columns of A, we can form the row space from the rows of A. The row space of A is the set of all linear combinations of the rows of A.

Since rows of an m×n matrix are vectors with n components, the row space lives in Rⁿ. We write it C(Aᵀ) — the column space of Aᵀ (the transpose), because transposing turns rows into columns.

Row space of A = C(Aᵀ) = all linear combinations of rows of A ⊆ Rⁿ

Here is the key fact about row operations: elementary row operations do not change the row space. When we perform Gaussian elimination — adding multiples of one row to another — the new rows are still combinations of the original rows, and vice versa. So the row space is preserved throughout elimination.

This is why the row-reduced form U of A has the same row space as A itself. The pivot rows of U span exactly the same space as the rows of A, even though they look different.

5. How Big Is a Subspace? Dimension

Every subspace has a 'size', measured by a number called its dimension. The dimension of a subspace is the number of independent vectors needed to span it — to reach every vector in it through combinations.

We will develop this idea fully in Section 3.4, but here is the key preview: the dimension of the column space of A equals the number of pivot columns in A after elimination. This number is called the rank of A.

dim(C(A)) = rank(A) = number of pivot columns

A 3×4 matrix with 2 pivot columns has column space of dimension 2 — a plane inside R³. The other two columns are combinations of the pivot columns and add no new directions.

The rank of A is both the dimension of C(A) and the dimension of C(Aᵀ). These two spaces can be very different in where they live (Rᵐ vs Rⁿ), but they always have the same dimension. This is a beautiful and non-obvious fact.

6. Why Subspaces Matter

Subspaces are not just abstract mathematics — they are the natural setting for understanding what a matrix does and does not do.

Every matrix A transforms vectors from Rⁿ to Rᵐ. The column space tells us the range of that transformation: which outputs are possible? The row space tells us which inputs actually matter: two inputs that differ only in the nullspace direction produce the same output (we meet the nullspace in Section 3.2). Together, these spaces reveal the complete geometry of the transformation.

Think of it this way: when you multiply Ax, the result always lands in C(A) — no matter what x you choose. The column space is the 'target range' of the matrix. Understanding C(A) tells you everything about which problems Ax = b can be solved.

In data science and machine learning, the column space of a data matrix represents the space of all outcomes that the model can explain. Vectors outside C(A) are the 'residuals' — the errors that the model cannot capture. Minimizing those residuals is the idea behind least squares, which builds directly on the column space.

References & Further Reading

Gilbert Strang. Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press, 2016. (Chapter 3.1)

3Blue1Brown. Essence of Linear Algebra, Chapter 2: Linear Combinations, Span, and Basis Vectors. YouTube, 2016.

Khan Academy. Vector spaces and subspaces unit. khanacademy.org/math/linear-algebra

End of Blog 4 — Linear Algebra Series