XGBoost
XGBoost (Extreme Gradient Boosting) is one of the most powerful and widely used machine learning algorithms for structured/tabular data. It dominated Kaggl...
Gradient Boosting with XGBoost
XGBoost (Extreme Gradient Boosting) is one of the most powerful and widely used machine learning algorithms for structured/tabular data. It dominated Kaggle competitions for years and remains a top choice for industry practitioners.
Gradient boosting builds an ensemble of decision trees sequentially. Each new tree corrects the errors of the previous trees. XGBoost adds regularization (L1 and L2), parallel tree construction, and hardware-level optimizations that make it significantly faster and more accurate than earlier boosting methods.
Install with pip install xgboost. XGBoost integrates with the Scikit-learn API: import xgboost as xgb from sklearn.metrics import accuracy_score
model = xgb.XGBClassifier( n_estimators=200, max_depth=6, learning_rate=0.1, subsample=0.8, colsample_bytree=0.8, use_label_encoder=False, eval_metric='logloss', random_state=42 ) model.fit(X_train, y_train) y_pred = model.predict(X_test) print(accuracy_score(y_test, y_pred))
Key hyperparameters to tune: - n_estimators: number of trees (more is better but slower; use early stopping) - max_depth: depth of each tree (3-6 is typical; deeper trees overfit) - learning_rate (eta): shrinks each tree's contribution (lower = more robust, needs more trees) - subsample: fraction of training rows used per tree (reduces overfitting) - colsample_bytree: fraction of features used per tree
Early stopping prevents overfitting by halting training when validation performance stops improving: model = xgb.XGBClassifier(n_estimators=1000, early_stopping_rounds=50) model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
XGBoost also supports native DMatrix format for speed: dtrain = xgb.DMatrix(X_train, label=y_train) dtest = xgb.DMatrix(X_test)
Practical tip: Start with learning_rate=0.1 and n_estimators=100. Use early stopping with a validation set to find the optimal number of trees, then reduce the learning rate and scale up n_estimators for a final model.
Exam tip: XGBoost handles missing values natively by learning the best direction to send missing data at each split. You do not need to impute before passing data to XGBoost.