Beginner Explanation
Imagine you are trying to find the best flavor of ice cream at a new shop, but tasting each one takes time and effort. Instead of tasting every flavor, you ask your friends which ones they liked best and use that information to guess which flavors might be the best. Each time you taste a new flavor, you learn more about what you like. Bayesian Optimization works similarly: it helps you figure out the best option (like the best ice cream flavor) by using past experiences (previous tastings) to make smart guesses about where to look next, minimizing the effort needed to find the best choice.Technical Explanation
Bayesian Optimization is a sequential design strategy for optimizing an unknown objective function that is costly to evaluate. It uses a surrogate model (often a Gaussian Process) to model the objective function and a utility function (like Expected Improvement) to decide where to sample next. The process involves: 1) Initializing with a few sample points, 2) Fitting the surrogate model to these points, 3) Using the utility function to find the next sampling point, and 4) Evaluating the objective function at this point. This loop continues until a stopping criterion is met. Here’s a simple Python example using the `GPyOpt` library: “`python import GPy import GPyOpt def objective_function(x): return (x – 2) ** 2 bounds = [{‘name’: ‘var1’, ‘type’: ‘continuous’, ‘domain’: (0, 4)}] optimizer = GPyOpt.methods.BayesianOptimization(f=objective_function, domain=bounds) optimizer.run_optimization(max_iter=10) print(optimizer.x_opt, optimizer.fx_opt) “`Academic Context
Bayesian Optimization is rooted in Bayesian statistics and machine learning. It was notably popularized in the context of hyperparameter tuning in machine learning models. The key mathematical foundation is the use of a probabilistic model (often a Gaussian Process) to define a prior over the objective function and update this prior with observed data to form a posterior. Key papers include ‘Practical Bayesian Optimization of Machine Learning Algorithms’ by Snoek et al. (2012), which discusses its application in tuning hyperparameters, and ‘Bayesian Optimization via Local Penalization’ by Kandasamy et al. (2017), which introduces more efficient sampling strategies. The mathematical formulation involves concepts like acquisition functions, which guide the search for optimal values by balancing exploration and exploitation.Code Examples
Example 1:
import GPy
import GPyOpt
def objective_function(x):
return (x - 2) ** 2
bounds = [{'name': 'var1', 'type': 'continuous', 'domain': (0, 4)}]
optimizer = GPyOpt.methods.BayesianOptimization(f=objective_function, domain=bounds)
optimizer.run_optimization(max_iter=10)
print(optimizer.x_opt, optimizer.fx_opt)Example 2:
import GPy
import GPyOpt
def objective_function(x):
return (x - 2) ** 2Example 3:
import GPyOpt
def objective_function(x):
return (x - 2) ** 2
Example 4:
def objective_function(x):
return (x - 2) ** 2
bounds = [{'name': 'var1', 'type': 'continuous', 'domain': (0, 4)}]
optimizer = GPyOpt.methods.BayesianOptimization(f=objective_function, domain=bounds)View Source: https://arxiv.org/abs/2511.16340v1