What is FAISS?
- Faiss is a library for efficient similarity search and clustering of dense vectors.
- Optimized for searching through millions or billions of high-dimensional vectors quickly
- Faiss contains several methods for similarity search.
Two main approaches (that we will be focusing on):
IndexFlatL2: Exact L2 matching but faster than manual implementation
IndexIVF (Inverted file): Clusters similar features together, only searches relevant clusters
IndexFlatL2 is similar to our cosine distance matching from the previous tutorial.
You may not notice any speed difference both, especially for smaller datasets.
For large datasets, IndexFlatL2 will still be slow since it does exhaustive search.
That's where IndexIVF becomes valuable
(by reducing the number of comparisons needed through clustering.)
References:
https://arxiv.org/abs/1702.08734
https://arxiv.org/abs/2401.08281
Python code available here: https://github.com/bnsreenu/python_for_microscopists/tree/master/349-Understanding%20FAISS