paper: https://arxiv.org/abs/1409.3215
Test of Time Paper Awards for NeurIPS 2024 https://blog.neurips.cc/2024/11/27/announcing-the-neurips-2024-test-of-time-paper-awards/
00:00 Award.
01:34 What we did
02:04 What we got right
03:20 What we got right: Autoregressive Models
04:17 What we got wrong: the LSTM
05:00 Early distributed training: Parallelization
06:00 The core idea
07:18 The age of Pre-Training
08:20 Pre-Training as we know it will end
08:45 The age of pre-training will end due to that the data is not growing
09:00 What comes next?
09:53 What comes next? Example from nature
12:45 What comes next? The long term
16:30 The end of talk: Q&A
16:46 Question 1: Now in 2024, are there other biological structures that are part of human cognition that you think are worth exploring in a similar way or that you're interested in anyway?
18:16 Question 2: Hallucinations in today's models
20:31 Question 3: How do you create the right incentive mechanisms for humanity to actually create it in a way that gives it the freedoms that we have as homosapiens?
22:23 Question 4: Do you think LLMs generalize multi-hop reasoning out of distribution?