Gradient Descent
In plain terms
The optimization method that trains models by stepping toward lower error.
Going deeper
Gradient descent measures how changing each parameter affects the error, then nudges parameters in the direction that reduces it, repeating until the model converges. It's the workhorse of training.
Under the hood
Variants (SGD, mini-batch, Adam, RMSProp) trade speed and stability; learning rate is the key hyperparameter, and non-convex loss surfaces have local minima/saddle points.
Examples
Training neural nets, logistic/linear regression, Adam optimizer, learning-rate schedules.
Think of it like...
Like descending a foggy mountain by always stepping in the steepest downhill direction until you reach the valley floor.