Find out how to improve training beyond the “vanilla” gradient descent algorithm

In my last post, we discussed how you may improve the performance of neural networks through hyperparameter tuning:

This can be a process whereby one of the best hyperparameters reminiscent of learning rate and variety of hidden layers are “tuned” to seek out probably the most optimal ones for our network to spice up its performance.

Unfortunately, this tuning process for giant deep neural networks (deep learning) is painstakingly slow. One method to improve upon that is to make use of faster optimisers than the normal “vanilla” gradient descent method. On this post, we are going to dive into the most well-liked optimisers and variants of gradient descent that may enhance the speed of coaching and in addition convergence and compare them in PyTorch!

Before diving in, let’s quickly brush up on our knowledge of gradient descent and the speculation behind it.

The goal of gradient descent is to update the parameters of the model by subtracting the gradient (partial derivative) of the parameter with respect to the loss function. A learning rate, α, serves to manage this process to make sure updating of the parameters occurs on an inexpensive scale and doesn’t over or undershoot the optimal value.

• θ are the parameters of the model.
• J(θ) is the loss function.
• ∇J(θ) is the gradient of the loss function. ∇ is the gradient operator, also generally known as nabla.
• α is the training rate.

I wrote a previous article on gradient descent and the way it really works if you would like to familiarise yourself a bit more about it: