Speaker : Iraklis Psaroudakis
Modern-day observability & security demand fast-paced searches in ever-increasing data volumes. Elasticsearch (ES), at the core of the Elastic Stack, is a leading distributed AI search and analytics engine. ES has been stateful so far, using primarily the disk to store data. In this presentation, we show the technical design of the new ES Serverless (ES3) mode, that achieves unparalleled scalability without any administration burden. We focus on the basic premise of decoupling compute from storage, and offloading data to an affordable highly available object store (e.g., S3), while supporting the same APIs and read-after-write semantics. Specifically, we show why and how we simplify node tiers to just two: indexing and search. We describe how we use batched compound commits to wrap Lucene files onto the object store, how the translog is buffered on the object store for recovery, and how refresh and search semantics operate.