Welcome to our deep dive into the world of optimizers! In this video, we'll explore the crucial role that optimizers play in machine learning and deep learning. From Stochastic Gradient Descent to Adam, we cover the most popular algorithms, how they work, and when to use them.
🔍 What You'll Learn:
Basics of Optimization - Understand the fundamentals of how optimizers work to minimize loss functions
Gradient Descent Explained - Dive deep into the most foundational optimizer and its variants like SGD, Momentum, and Nesterov Accelerated Gradient
Advanced Optimizers - Get to grips with Adam, RMSprop, and AdaGrad, learning how they differ and their advantages
Intuitive Math - Unveil the equations for each optimizer and learn how it stands out from the others
Real World Benchmarks - See real world experiments from papers in domains ranging from computer vision to reinforcement learning to see how these optimizers fare against each other
🔗 Extra Resources:
3Blue1Brown - https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi
Artem Kirsanov - https://www.youtube.com/watch?v=SmZmBKc7Lrs
📌 Timestamps:
0:00 - Introduction
1:17 - Review of Gradient Descent
5:37 - SGD w/ Momentum
9:26 - Nesterov Accelerated Gradient
10:55 - Root Mean Squared Propagation
13:59 - Adaptive Gradients (AdaGrad)
14:47 - Adam
18:12 - Benchmarks
22:01 - Final Thoughts
Stay tuned and happy learning!