Unsupervised pre-training for few-shot learning, vol. 2: reconstruction-based methods
For more information about Stanford's Artificial Intelligence programs visit: https://stanford.io/ai
To follow along with the course, visit:
https://cs330.stanford.edu/
To view all online courses and programs offered by Stanford, visit: http://online.stanford.edu
Chelsea Finn
Computer Science, PhD
Plan for Today
Recap
- Problem formulation
- Contrastive learning
Reconstruction-based unsupervised pre-training
- Why reconstruction?
- Autoencoders
- Masked autoencoders: BERT, MAE
- Autoregressive models: GPT, Flamingo
Goals for by the end of lecture:
- Familiarize you with widely-used methods for unsupervised pre-training
- Introduce methods for efficient fine-tuning of pre-trained models
- Prepare you for HW3