Implement Llama 3 From Scratch - PyTorch

We will implement Llama 3 from scratch using PyTorch and generate a sample text with it! This is the annotated Llama. GitHub below ↓ Want to support the channel? Hit that like button and subscribe! Since all Llama models are built on top of each other, you can also use this for your other Llama implementations. GitHub Link of the Code https://github.com/uygarkurt/Llama-3-PyTorch Llama 3 is introduced in the paper: "The Llama 3 Herd of Models" https://arxiv.org/abs/2407.21783 Meta Website to Download Llama 3 Weights https://www.llama.com/llama-downloads/ Meta's Official Implementation https://github.com/meta-llama/llama3 What should I implement next? Let me know in the comments! 00:00:00 Introduction 00:00:56 Download Llama 3 Weights 00:01:34 Llama 3 Architecture Overview 00:04:35 Imports 00:06:46 Hyperparameter Definitions 00:13:40 RMSNorm & RoPE 00:14:19 Feed Forward SwiGLU 00:20:29 Grouped Query Attention (GQA) With KV Cache 00:38:22 Transformer Block 00:44:40 Transformer (Llama Model) 00:56:30 Text Generation 01:00:20 Outro Buy me a coffee! ☕️ https://ko-fi.com/uygarkurt