Gradient Descent

Understand it deeply

What is gradient descent, really?

A method for repeatedly adjusting model parameters so the error gets smaller.

Training does not directly search for the correct answer. It searches for model parameters that make a loss function as small as possible. Gradient descent is the update method for those parameters.

Think of walking down a mountain

Imagine standing in fog on a mountain and trying to reach the lowest point. You cannot see the entire terrain, but you can feel which nearby direction slopes down most steeply. Take a small step, check again, and repeat until you approach the valley floor.

The three-step loop

1. Error (loss)

   Make a prediction with the current parameters and compare it with the real answer. A lower loss means less error.

2. Gradient (direction)

   Calculate the slope of the loss at the current point. It points uphill fastest, so we travel in the opposite direction.

3. Update the parameters

   Use the learning rate to choose a step size, move opposite the gradient, then repeat until the loss is small enough.

What the formula says

The formula simply turns “read the slope, then take a small step” into math.

w_new = w_old − η × ∂Loss/∂w

• w — Model parameter

  A number inside the model that can be adjusted.

• ∂Loss/∂w — Gradient

  How the loss changes when a parameter changes slightly.

• η — Learning rate

  How far each step goes; too large can overshoot, too small learns slowly.

What happens during neural-network training?

A large neural network repeats the same loop millions of times: feed in training data, predict, calculate loss, use backpropagation to calculate gradients, update parameters with gradient descent, and predict again. Better parameters gradually lead to better predictions.

Common misunderstandings

Does gradient descent find the correct answer?

No. It adjusts internal parameters to make the loss as small as possible. Lower loss often improves predictions, but the two are not identical claims.

Why is backpropagation still needed?

Backpropagation efficiently calculates how each parameter affects the loss. Gradient descent uses those gradients to update the parameters.

Gradient descent follows the direction that reduces error fastest, takes small steps, and keeps adjusting parameters until it approaches the valley floor.