Grid search
Grid search is a hyperparameter tuning method that exhaustively tries every combination of specified hyperparameter values and selects the combination that...
Systematic Hyperparameter Optimization
Grid search is a hyperparameter tuning method that exhaustively tries every combination of specified hyperparameter values and selects the combination that produces the best validation performance. It is straightforward to implement and guarantees that the optimal combination within the search space is found, but it scales poorly with the number of hyperparameters.
To run a grid search you define a grid: a dictionary mapping each hyperparameter to a list of candidate values. For a support vector machine you might specify candidate values for the regularization parameter C as [0.01, 0.1, 1, 10, 100] and for the kernel parameter gamma as [0.001, 0.01, 0.1, 1]. Grid search trains and evaluates a model for every combination — in this example, 25 models. The combination with the best cross-validated performance is selected.
The computational cost of grid search grows exponentially with the number of hyperparameters. Adding a third hyperparameter with five candidate values multiplies the number of experiments by five. For complex models like neural networks with many hyperparameters, grid search becomes impractical.
Random search addresses this by sampling random combinations from the hyperparameter space rather than trying all of them. Empirically, random search often finds equally good configurations in far fewer trials, because most hyperparameters have little effect on performance and only a few matter significantly.
Bayesian optimization takes a smarter approach: it builds a probabilistic model of the relationship between hyperparameter values and validation performance, then uses this model to choose the next combination most likely to improve results. It is more efficient than both grid search and random search for expensive models.
A concrete example: grid searching a gradient boosting model across three hyperparameters with five values each requires 125 training runs. Using 5-fold cross validation, this means 625 model fits — a significant computational investment.
Exam tip: know the three main tuning approaches — grid search (exhaustive), random search (sampled), and Bayesian optimization (adaptive) — and when each is appropriate based on computational budget.