The Beautiful Math Behind Learning — Understanding the Perceptron's Update Rule

Machine-Learning tutorial · PySpark.in

1. Introduction: A Teacher Learning from Mistakes

Imagine you are a new teacher. On your first day, you know nothing. You stand in front of a classroom with two groups of students: Group A (wearing blue shirts) and Group B (wearing red shirts). Your job is to draw a line on the floor that separates the two groups.

On the first try, your line is completely wrong. Some blue students are on the red side, and some red students are on the blue side. So what do you do? You look at each student one by one. If a student is on the wrong side, you gently push the line so that student is now on the correct side.

You keep doing this — looking at every student, fixing the line when someone is wrong — until finally, everyone is on the correct side.

This is exactly how the Perceptron learns.

In the last blog, we met the Perceptron — the first machine that could "learn" by looking at data. In this blog, we will look at the Mathematical Coda from Chapter 2 of Why Machines Learn. The author, Anil Ananthaswamy, calls this a "beautiful proof." We will break it down into such simple steps that you will see why it is beautiful, even if you have never studied advanced math.

2. The Three-Step Recipe (The Algorithm)

The Perceptron follows a very simple recipe. There are no hidden tricks. Let us write it in plain English first, then in math.

Step 1: Start Blind

Set your "weight vector" w to zero. This means the machine starts with no opinion at all. It is like the teacher who has not yet drawn any line.

Step 2: Look at Every Student

Go through every data point x in your training set. For each student, check if they are on the wrong side of the line.

How do you check? You calculate the score: y (wᵀx).

If y (wᵀx) ≤ 0, the machine made a mistake. For example:

When there is a mistake, you update the line:

This is the magic rule. It says: "Take the old line, and nudge it by adding y times the student's position."

Step 3: Repeat Until Perfect

If you went through all the students and made zero mistakes, you are done! If not, go back to Step 2 and look at all the students again.

That is the entire algorithm. No calculus. No heavy formulas. Just: look, check, fix, repeat.

3. Why Does the Fix Work? (The Math Coda Explained)

Now comes the beautiful part. The book asks a simple question: How do we know that updating w actually helps? Maybe we are just moving the line randomly and making things worse!

Let us check. After an update, we want to see if the same student x is now classified better. We calculate the new score:

Put the update rule inside:

If we open the brackets (like opening a box), we get two pieces:

Let us look at the second piece: y² (xᵀ x).

Therefore, the second piece is always positive or zero. We are adding a positive number to the old score!

This means:

New Score = Old Score + (something positive)

So the new score is bigger than the old score. If the old score was negative (a mistake), the new score is less negative — it moved upward, closer to being positive. The line is bending in the right direction for that student.

In simple words: Every time the machine makes a mistake and updates, it is not guessing. It is mathematically guaranteed to push the score in the correct direction.

Look at the picture above. On the left, the green dashed line is wrong — the red student is on the wrong side. The machine says, "Mistake!" On the right, after applying the update rule, the blue solid line moves to correct the error. The student is now on the correct side.

4. The Promise: It Always Finishes

Here is another amazing fact from the book. The proof shows that the number of updates is always finite. The machine cannot keep making mistakes forever.

Why?

Imagine there is a perfect teacher in the room — let us call her w*. She knows the perfect line that separates all blue and red students perfectly. The book tells us to imagine w* as a unit vector (length = 1) pointing in the perfect direction.

Every time our machine makes a mistake and updates w, our w moves closer to the perfect w*. It is like walking toward a lighthouse in fog. You cannot see the lighthouse clearly, but every step you take is a step in the right direction.

Because you are getting closer and closer to a fixed point (the lighthouse), and because your steps have a minimum size, you must reach it after a finite number of steps. You cannot take infinite steps toward a fixed point if each step has a minimum size.

In the picture above, the green arrow is the perfect line w*. The black dot at the center is where we start (w = 0). With every update, the blue dots show our weight vector moving closer and closer to the perfect direction. The angle between our line and the perfect line keeps shrinking.

This is the heart of the convergence proof. The book calls it a "Mathematical Coda," but really, it is a promise: If the students can be separated by a straight line, the Perceptron will always find that line in a finite number of steps.

5. Try It Yourself: Pure Python Code

The best way to understand learning is to make the machine learn with your own hands. Below is a complete Perceptron written in pure Python — no NumPy, no sklearn, no external libraries at all. Just basic Python lists and loops.

This code follows the exact three steps from the book:

```

python

# ============================================

# PERCEPTRON FROM SCRATCH - Pure Python

# No numpy, no sklearn, no external libraries!

# ============================================

def dot_product(a, b):

"""Calculate dot product of two vectors."""

return sum(x * y for x, y in zip(a, b))

def vector_add(a, b):

"""Add two vectors."""

return [x + y for x, y in zip(a, b)]

def scalar_multiply(s, v):

"""Multiply vector v by scalar s."""

return [s * x for x in v]

def perceptron_train(data, labels, max_iterations=100):

"""

Train a perceptron using the update rule from the book.


data: list of feature vectors (each vector is a list of numbers)

labels: list of +1 or -1

max_iterations: safety limit to prevent infinite loops

"""

# Step 1: Initialize weight vector to zero

num_features = len(data[0])

w = [0.0] * num_features


print("Step 1: Starting with weights =", w)

print("-" * 50)


for iteration in range(max_iterations):

updates_done = 0

print(f"\n--- Pass {iteration + 1} over all data ---")


# Step 2: For each data point

for i, (x, y) in enumerate(zip(data, labels)):

# Calculate prediction: w^T * x

score = dot_product(w, x)


# Step 2a: If y * (w^T x) <= 0, the prediction is wrong

if y * score <= 0:

# Update rule: w_new = w_old + y * x

update = scalar_multiply(y, x)

w = vector_add(w, update)

updates_done += 1

print(f" Point {i}: x={x}, y={y}, score={score:.2f}")

print(f" -> MISTAKE! Updating weights: w = w + ({y})*{x}")

print(f" -> New weights: {[round(v, 2) for v in w]}")

else:

print(f" Point {i}: x={x}, y={y}, score={score:.2f} -> OK")


# Step 3: If no updates, we are done!

if updates_done == 0:

print("\n" + "=" * 50)

print(f"SUCCESS! No mistakes in pass {iteration + 1}.")

print(f"Final weights: {[round(v, 2) for v in w]}")

print("=" * 50)

return w


print(f"\nPass {iteration + 1} complete. Mistakes fixed: {updates_done}")


print("Reached max iterations.")

return w

def perceptron_predict(w, x):

"""Predict label (+1 or -1) for a new point."""

score = dot_product(w, x)

return 1 if score > 0 else -1

# ============================================

# EXAMPLE: Learning to separate two groups

# ============================================

# Simple 2D data: [x1, x2, bias_term]

# Bias term is always 1 (trick to include threshold in weight vector)

training_data = [

[1, 1, 1], # Positive example

[2, 2, 1], # Positive example

[3, 1, 1], # Positive example

[1, 3, 1], # Negative example

[2, 3, 1], # Negative example

]

labels = [1, 1, 1, -1, -1]

print("Training the Perceptron...")

print("Data points:", training_data)

print("Labels:", labels)

print()

final_w = perceptron_train(training_data, labels)

# Test on new points

print("\n--- Testing ---")

test_points = [

[2, 1, 1], # Should be positive

[1, 4, 1], # Should be negative

]

for point in test_points:

pred = perceptron_predict(final_w, point)

print(f"Point {point[:2]} -> Prediction: {'Positive' if pred == 1 else 'Negative'}")

```

Run this code on any Python environment (even online). Watch the print statements. You will see the machine start with [0, 0, 0], make mistakes, fix them, and finally say "SUCCESS!" It is like watching a child learn to walk — wobbly at first, then steady.

6. Summary in Simple Words

Let us wrap up what we learned today:

  1. The Perceptron starts blind (w = 0).
  2. It looks at every data point. If a point is on the wrong side, it updates the line using w new = w old + yx
  3. This update is mathematically guaranteed to improve the score for that point, because we are always adding a positive amount (y²xᵀx) to push the score upward.
  4. Because each update moves w closer to the perfect w*, the algorithm cannot run forever. It always finishes after a finite number of steps.
  5. If the data can be separated by a straight line, the Perceptron will find it. This is not hope. This is mathematical proof.

That is the beauty of the Mathematical Coda. It takes a simple idea — "fix your mistakes" — and proves, with nothing more than basic algebra, that the machine will learn.

7. Reference

Anil Ananthaswamy, Why Machines Learn: The Elegant Math Behind Modern AI (2024)

Kilian Weinberger's 2018 Cornell lecture

More Machine-Learning tutorials

All tutorials · Try the free PySpark compiler · Practice challenges