Back to Study Material Machine Learning

Hyperparameters

Hyperparameters are configuration values that control how a machine learning algorithm learns. Unlike model parameters — which are learned from data duri...

Hyperparameters

Settings That Shape the Learning Process

Hyperparameters are configuration values that control how a machine learning algorithm learns. Unlike model parameters — which are learned from data during training — hyperparameters are set before training begins and remain fixed throughout. Choosing good hyperparameters is critical to model performance and generalization.

The distinction is important. In a neural network, the weights connecting neurons are parameters learned through gradient descent. The number of layers, the number of neurons per layer, the learning rate, and the dropout rate are all hyperparameters set by the practitioner. They determine the capacity of the model and the dynamics of the learning process.

Learning rate is one of the most important hyperparameters in any gradient descent-based model. Set too high, the optimizer overshoots the minimum and may diverge entirely. Set too low, training proceeds extremely slowly and may get stuck in local minima. Most practitioners start with a standard value and use a learning rate schedule that reduces the rate during training.

Regularization strength is another key hyperparameter. In Ridge regression, the parameter lambda controls how strongly large coefficients are penalized. Low lambda leads to overfitting; high lambda leads to underfitting. The right value depends on the dataset and must be found empirically.

In tree-based models, maximum depth, minimum samples per leaf, and number of estimators in an ensemble are hyperparameters that control complexity. In K-nearest neighbors, k itself is a hyperparameter.

A concrete example: a random forest with default hyperparameters achieves 78 percent accuracy on a validation set. After tuning n_estimators, max_depth, and min_samples_split via grid search, accuracy improves to 85 percent on the same validation data.

Exam tip: be clear that hyperparameters are not learned from data — they are chosen by the practitioner using techniques like grid search, random search, or Bayesian optimization on the validation set.