In optimization, we often encounter expensive black-box problems with unknown problem structures. Bayesian Optimization (BO) is a popular, surrogate-assisted and thus sample-efficient approach for this setting. The BO pipeline itself is highly configurable with many different design choices regarding the initial design, surrogate model and acquisition function (AF). Unfortunately, our understanding of how to select suitable components for a problem at hand is very limited. In this work, we focus on the choice of the AF, whose main purpose it is to balance the trade-off between exploring regions with high uncertainty and those with high promise for good solutions. We propose Self-Adjusting Weighted Expected Improvement (SAWEI), where we let the exploration-exploitation trade-off self-adjust in a data-driven manner based on a convergence criterion for BO. On the BBOB functions of the COCO benchmark, our method performs favorably compared to handcrafted baselines and serves as a robust default choice for any problem structure. With SAWEI, we are a step closer to on-the-fly, data-driven and robust BO designs that automatically adjust their sampling behavior to the problem at hand.