Title: Understanding High-Dimensional Bayesian Optimization
Speaker: Leonard Papenmeier (https://leonard.papenmeier.io/)
Paper: https://arxiv.org/abs/2502.09198
Abstract:
Recent work reported that simple Bayesian optimization methods perform well for high-dimensional real-world tasks, seemingly contradicting prior work and tribal knowledge. In this talk, we identify fundamental challenges in high-dimensional Bayesian optimization (HDBO) and explain why recent methods succeed. Our analysis shows that two types of vanishing gradients caused by Gaussian process (GP) initialization schemes play a major role in the failures of high-dimensional Bayesian optimization and that methods that promote local search behaviors are better suited for the task. We discuss how a simple variant of maximum likelihood estimation of GP length scales achieves state-of-the-art performance on a comprehensive set of real-world applications by leveraging these insights and discuss whether HDBO can be considered solved.