Linear Equations

Linear-Algebra tutorial · PySpark.in

1. What Makes an Equation "Linear"?

Before diving into systems of equations, it is worth asking a simple question: what makes an equation linear in the first place? An equation is linear when every unknown appears only by itself, multiplied by a plain number (called a coefficient). No squares, no cubes, no products of two unknowns together.

So x - 2y = 1 is linear — both x and y appear alone, multiplied by their coefficients 1 and -2. But x² - 2y = 1 is not linear, because of the square on x. And xy = 1 is not linear either, because two unknowns are multiplied together. Linear equations are the simplest kind — and yet they are powerful enough to describe an enormous range of real-world problems, from electrical circuits to supply-and-demand models to the training of neural networks.

The central problem of linear algebra is to solve a system of linear equations. Those equations are linear, which means the unknowns are only multiplied by numbers — we never see x times y. Our first linear system is small, but you will see how far this idea leads. — Gilbert Strang

2. A System of Two Equations

The classic starting example from Chapter 2 of Strang's book involves two unknowns, x and y, and two equations that must both be satisfied simultaneously:

x - 2y = 1 (Equation 1)

3x + 2y = 11 (Equation 2)

We want to find the single pair of values (x, y) that makes both equations true at the same time. Because both equations are linear, each one — when drawn on a graph — produces a perfectly straight line. The solution, geometrically, is wherever those two lines cross.

The Row Picture: Lines Meeting at a Point

If we draw Equation 1 on the xy-plane, we get a straight line. The point (1, 0) is on this line because 1 - 2(0) = 1 ✓. So is the point (3, 1), because 3 - 2(1) = 1 ✓. The second equation 3x + 2y = 11 draws another straight line. Where they cross — the point (3, 1) — satisfies both equations at once. That crossing point is the solution.

The row picture shows two lines meeting at a single point — and that point is the solution. Every equation draws one line (or plane in higher dimensions), and the solution is where they all intersect.

The Column Picture: Combining Vectors

Now here is a completely different way to see the exact same problem. Instead of looking row by row, we look at the columns of the coefficient matrix. We rewrite the system as a vector equation:

x · [ 1] + y · [-2] = [ 1]

[ 3] [ 2] [11]

The question becomes: what multiples x and y of the two column vectors on the left will produce the right-hand side vector (1, 11)? With x = 3 and y = 1, we get 3·(1, 3) + 1·(-2, 2) = (3, 9) + (-2, 2) = (1, 11). The column picture turns equation-solving into finding the right linear combination of column vectors.

The column picture combines the column vectors on the left side to produce the vector b on the right side. This view connects directly to the concept of linear combinations — the cornerstone of all of linear algebra.

3. Packing It Into a Matrix: Ax = b

The two perspectives — row picture and column picture — are two windows onto the same object: the matrix equation Ax = b. We gather the coefficients into a matrix A, the unknowns into a vector x, and the right-hand sides into a vector b:

[ 1 -2 ] · [ x ] = [ 1 ]

[ 3 2 ] [ y ] [ 11 ]

This compact notation is not merely aesthetic. It lets us bring all the machinery of linear algebra to bear on any system, no matter how large. Two equations or two thousand equations, the idea is always the same: Ax = b.

A is the coefficient matrix. It stores all the coefficients from the left sides of the equations. x is the vector of unknowns. b is the vector of right-hand side values. The entire system is summarised in the single equation Ax = b.

4. Solving by Elimination

The most systematic method for solving a linear system is elimination. The core idea is to combine equations to cancel out unknowns one at a time, reducing the system to a simpler triangular form that can be solved from the bottom up.

For our two-equation system, we multiply Equation 1 by 3 and subtract it from Equation 2 to eliminate x:

Equation 2 - 3 × Equation 1:

(3x + 2y) - 3(x - 2y) = 11 - 3(1)

8y = 8 ---> y = 1

Back-substitute into Equation 1:

x - 2(1) = 1 ---> x = 3

This process of using the value of y to find x is called back substitution — once you have the bottom unknown, you substitute it back up the chain. Strang lists four key steps to this process for any size system: elimination produces a triangular matrix U, back substitution solves that triangular system, and in matrix language the original A is factored into A = LU, a product of a lower triangular and an upper triangular matrix. LU factorisation is how computers efficiently solve enormous linear systems.

5. The Three Possible Outcomes

Not every system of linear equations has a solution, and some have infinitely many. The outcome depends entirely on the geometry of the situation — how the lines, planes, or hyperplanes are arranged.

Unique Solution

When n equations in n unknowns have exactly one solution, the lines (or planes) meet at exactly one point in space. The matrix A is invertible, and the solution is x = A⁻¹b. This is the generic, well-behaved case.

No Solution

If two equations describe parallel lines (same slope, different intercept), they never meet. No point satisfies both equations simultaneously. The system is inconsistent. In matrix language, the target vector b lies outside the column space of A — it is simply unreachable.

Infinitely Many Solutions

If the two equations actually describe the same line (one is a multiple of the other), then every point on that line is a solution. The matrix is singular — its columns are linearly dependent. There is a whole line (or plane) of solutions rather than a unique answer.

Geometry and algebra tell the same story: unique solution means lines cross at one point and A is invertible. No solution means lines are parallel and b is out of reach. Infinite solutions means the lines coincide and A is singular.

6. The Solution Formula: x = A⁻¹b

For invertible matrices, the solution has a beautifully compact form. If Ax = b and A is invertible, multiply both sides on the left by A⁻¹:

A⁻¹ · A · x = A⁻¹ · b

I · x = A⁻¹ · b

x = A⁻¹ · b

For every vector b, there is exactly one solution x = A⁻¹b. The inverse matrix literally undoes the transformation that A applied. It is the matrix equivalent of dividing both sides of a simple scalar equation by its coefficient.

In practice, for large systems, computing A⁻¹ directly is inefficient. Elimination (which gives LU factorisation) is almost always faster. But the formula x = A⁻¹b tells us something important conceptually: the solution exists and is unique whenever A is invertible, and it depends linearly on b.

References & Further Reading

Gilbert Strang. Introduction to Linear Algebra, 5th ed. Wellesley-Cambridge Press, 2016. (Chapter 2.1: Vectors and Linear Equations)

3Blue1Brown. Essence of Linear Algebra, Chapter 7: Inverse Matrices, Column Space, Null Space. YouTube, 2016.

Khan Academy. Systems of equations unit. khanacademy.org/math/algebra

End of Blog 2 — Linear Algebra Series