Last Minute Notes Machine Learning
This section condenses the most frequently tested machine learning formulas into a rapid-revision format. Each formula is accompanied by the context in whi...
Machine Learning Formula Sheet: Critical Equations for GATE
This section condenses the most frequently tested machine learning formulas into a rapid-revision format. Each formula is accompanied by the context in which it appears in GATE DA, so you can immediately recognize the question type and retrieve the correct equation.
Linear regression closed form: w = (X^T X)^(-1) X^T y. Ridge regression solution: w = (X^T X + lambda I)^(-1) X^T y. Logistic regression prediction: P(y=1|x) = 1 / (1 + exp(-w^T x)). Binary cross-entropy loss: L = -[y log(p) + (1-y) log(1-p)] where p is the predicted probability.
Information gain: IG(S, A) = H(S) - sum over values v of [|S_v|/|S| * H(S_v)], where H is entropy. Entropy: H(S) = -sum over classes c of [p_c * log2(p_c)]. Gini impurity: G(S) = 1 - sum over classes c of [p_c^2]. These three formulas characterize decision tree splitting criteria and are frequently substituted into small examples in GATE.
K-means objective: minimize sum over clusters k of [sum over points x in cluster k of ||x - mu_k||^2], where mu_k is the centroid of cluster k. EM algorithm for Gaussian mixtures alternates between computing posterior responsibilities and updating parameters. PCA: find eigenvectors of (1/n) X^T X; sort by decreasing eigenvalue.
Precision = TP/(TP+FP); Recall = TP/(TP+FN); F1 = 2PR/(P+R). Accuracy = (TP+TN)/(TP+TN+FP+FN). These definitions are given consistently in GATE; the challenge is applying them to a confusion matrix presented in the question.
Bias-variance decomposition: E[(y - f_hat(x))^2] = Bias^2(f_hat(x)) + Var(f_hat(x)) + sigma^2, where sigma^2 is irreducible noise. Regularization reduces variance at the cost of increased bias.
Quick tip: in any GATE ML question involving a table of TP, TN, FP, FN values, compute all four fundamental metrics immediately before reading the question options. This prevents calculation errors under time pressure and often reveals the answer before you read the choices.
Overfitting, Regularization, and Validation Strategies
Overfitting occurs when a model learns the training data so well that it captures noise and fails to generalize. Recognizing overfitting and selecting appropriate remedies is a key conceptual skill tested in GATE DA. The telltale sign is a large gap between training performance and validation performance.
Regularization penalizes model complexity to prevent overfitting. L2 regularization (ridge) adds lambda * sum(w_i^2) to the loss function. L1 regularization (lasso) adds lambda * sum(|w_i|). The regularization strength lambda controls the trade-off: lambda = 0 gives no regularization; large lambda drives weights toward zero, increasing bias. The optimal lambda is found by cross-validation.
Dropout is a regularization technique for neural networks where a random fraction p of neurons is set to zero during each training forward pass. This prevents neurons from co-adapting and forces the network to learn redundant representations. At test time, all neurons are active but their weights are scaled by (1 - p). Batch normalization normalizes activations within a mini-batch to have mean 0 and variance 1, which reduces internal covariate shift and allows higher learning rates.
Early stopping monitors validation loss during training and stops training when validation loss begins to increase, even if training loss continues to decrease. This implicitly regularizes by limiting the number of gradient steps. It requires setting aside a validation set separate from the test set.
Data augmentation artificially increases training set size by applying label-preserving transformations to existing samples. For images, common augmentations include rotation, flipping, cropping, and color jitter. For text, synonym replacement and back-translation are used. Augmentation reduces overfitting and improves robustness.
GATE tip: a question might present training accuracy of 99 percent and test accuracy of 72 percent and ask for the likely cause. The answer is overfitting. Follow-up options might ask for solutions; valid answers include adding more training data, reducing model complexity, increasing regularization, or using cross-validation to tune hyperparameters. Eliminate options like increasing learning rate or adding more layers, which would worsen overfitting.
Clustering, Dimensionality Reduction, and Feature Engineering
Dimensionality reduction and feature engineering are practical skills tested in GATE DA with increasing frequency as the exam matures. These techniques bridge raw data and model inputs, and understanding their mathematical basis enables you to answer both formula and scenario questions.
PCA captures maximum variance in orthogonal directions. The first principal component is the direction that maximizes the variance of the projected data. It is the eigenvector corresponding to the largest eigenvalue of the covariance matrix. The proportion of variance explained by all k selected components is sum(lambda_1 to lambda_k) / sum(all lambdas). Scree plots visualize eigenvalues and help choose k by identifying the elbow where variance explained begins to level off.
t-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear dimensionality reduction technique for visualization. It models pairwise similarities as probabilities in both high and low dimensions and minimizes their KL divergence. t-SNE preserves local structure well but distorts global distances. It is computationally expensive and not suitable for new data points without retraining.
Feature scaling is essential for distance-based and gradient-based algorithms. Min-max scaling transforms features to the range [0,1]: x_scaled = (x - x_min) / (x_max - x_min). Standard scaling (z-score normalization) transforms to mean 0 and standard deviation 1: x_scaled = (x - mean) / std. StandardScaler is appropriate for algorithms that assume zero mean and unit variance, such as PCA and logistic regression with L2 regularization.
Feature selection reduces the number of features by removing irrelevant or redundant ones. Filter methods rank features by statistical metrics such as correlation or mutual information and select the top k. Wrapper methods train a model on subsets of features and select the subset with the best cross-validation performance. Embedded methods perform selection during training, such as lasso regression naturally zeroing out unimportant feature coefficients.
Worked example: if a dataset has 10 features and PCA is applied, and the eigenvalues are 5, 3, 1, and six others each less than 0.2, then the first two components explain (5+3)/(5+3+1+6*0.2) = 8/10.2 ≈ 78 percent of the variance. This type of proportion calculation appears regularly in GATE ML questions.