LoRA - Explained!

A parameter efficient fine tuning technique that makes use of a low rank adapter to (1) reduce storage required per task by decreasing the number of trainable parameters added to the network per task (2) remove inference latency ensuring the stored parameters are applied to the existing network architecture instead of adding more RESOURCES [1 📚] Paper that introduces LoRA: https://arxiv.org/pdf/2106.09685 [2 📚] Paper that introduced the additive adapter: https://arxiv.org/abs/2312.12148 [3 📚] Other PEFT techniques: https://arxiv.org/pdf/2312.12148 [4 📚] My video on Parameter Efficient Fine Tuning (PEFT): https://youtu.be/HcVtpLAGMXo [5 📚] Neural networks tend to be more complex than required, hence we can make use of a "low rank" adapter without losing performance accuracy: https://arxiv.org/pdf/1804.08838 ABOUT ME ⭕ Subscribe: https://www.youtube.com/c/CodeEmporium?sub_confirmation=1 📚 Medium Blog: https://medium.com/@dataemporium 💻 Github: https://github.com/ajhalthor 👔 LinkedIn: https://www.linkedin.com/in/ajay-halthor-477974bb/ PLAYLISTS FROM MY CHANNEL ⭕ Deep Learning 101: https://www.youtube.com/playlist?list=PLTl9hO2Oobd_NwyY_PeSYrYfsvHZnHGPU ⭕ Natural Language Processing 101: https://www.youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE ⭕ Reinforcement Learning 101: https://youtube.com/playlist?list=PLTl9hO2Oobd9kS--NgVz0EPNyEmygV1Ha&si=AuThDZJwG19cgTA8 Natural Language Processing 101: https://youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE&si=LsVy8RDPu8jeO-cc ⭕ Transformers from Scratch: https://youtube.com/playlist?list=PLTl9hO2Oobd_bzXUpzKMKA3liq2kj6LfE ⭕ ChatGPT Playlist: https://youtube.com/playlist?list=PLTl9hO2Oobd9coYT6XsTraTBo4pL1j4HJ MATH COURSES (7 day free trial) 📕 Mathematics for Machine Learning: https://imp.i384100.net/MathML 📕 Calculus: https://imp.i384100.net/Calculus 📕 Statistics for Data Science: https://imp.i384100.net/AdvancedStatistics 📕 Bayesian Statistics: https://imp.i384100.net/BayesianStatistics 📕 Linear Algebra: https://imp.i384100.net/LinearAlgebra 📕 Probability: https://imp.i384100.net/Probability OTHER RELATED COURSES (7 day free trial) 📕 ⭐ Deep Learning Specialization: https://imp.i384100.net/Deep-Learning 📕 Python for Everybody: https://imp.i384100.net/python 📕 MLOps Course: https://imp.i384100.net/MLOps 📕 Natural Language Processing (NLP): https://imp.i384100.net/NLP 📕 Machine Learning in Production: https://imp.i384100.net/MLProduction 📕 Data Science Specialization: https://imp.i384100.net/DataScience 📕 Tensorflow: https://imp.i384100.net/Tensorflow CHAPTERS 0:00 Introduction 1:49 Pass 1: Low Rank Matrices 8:00 Quiz 1 8:52 Pass 2: Adapters 16:38 Quiz 2 17:47 Pass 3: Low Rank Adapters 26:37 Quiz 3 27:54 Summary