In this video, we dive into Large Concept Models (LCMs), an innovative architecture from a recent Meta paper titled: "Large Concept Models: Language Modeling in a Sentence Representation Space".
Unlike Large Language Models (LLMs), LCMs operate on higher-level semantics, processing concepts instead of tokens, making them more akin to human reasoning, and possibly positioning them as a future rival to the current token-based LLM architecture.
Throughout the video, we explain what reasoning in the abstract concepts space means, and explore the Large Concept Models architecture, diving into few architecture options.
Specifically, two LCMs are based on Diffusion Models, so we provide a brief reminder about Diffusion Models and then explain the Diffusion-based LCMs.
Paper - https://arxiv.org/abs/2412.08821
Code - https://github.com/facebookresearch/large_concept_model
Written review - https://aipapersacademy.com/large-concept-models/
Large Concept Models resembles to Joint Embedding Predictive Architecture (JEPA) in the sense of information prediction in an abstract representation space. We've covered previous JEPA paper thoroughly here:
I-JEPA - https://aipapersacademy.com/i-jepa-a-human-like-computer-vision-model/
V-JEPA - https://aipapersacademy.com/v-jepa/
-----------------------------------------------------------------------------------------------
✉️ Join the newsletter - https://aipapersacademy.com/newsletter/
👍 Please like & subscribe if you enjoy this content
Become a patron - https://www.patreon.com/aipapersacademy
The video was edited using VideoScribe - https://tidd.ly/44TZEiX
-----------------------------------------------------------------------------------------------
Chapters:
0:00 Introduction
1:25 Concepts vs Tokens
3:08 LCM High-Level Architecture
5:31 Base-LCM
6:49 Diffusion-Based LCM
9:31 Results