Back to Study Material Python

LightGBM

LightGBM (Light Gradient Boosting Machine) is Microsoft's answer to XGBoost. It uses two novel techniques, Gradient-based One-Side Sampling (GOSS) and Excl...

LightGBM

Fast Gradient Boosting with LightGBM

LightGBM (Light Gradient Boosting Machine) is Microsoft's answer to XGBoost. It uses two novel techniques, Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), to train faster and use less memory, especially on large datasets with millions of rows.

The key architectural difference: XGBoost grows trees level-by-level (breadth-first), while LightGBM grows trees leaf-by-leaf (best-first). Leaf-wise growth finds the split that reduces loss the most, achieving better accuracy with fewer trees, but can overfit on small datasets if max_depth is not constrained.

Install with pip install lightgbm. LightGBM also provides a Scikit-learn compatible API: import lightgbm as lgb

model = lgb.LGBMClassifier( n_estimators=500, num_leaves=31, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, random_state=42 ) model.fit( X_train, y_train, eval_set=[(X_val, y_val)], callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)] )

Key hyperparameters specific to LightGBM: - num_leaves: maximum number of leaves per tree (most important parameter; default 31) - min_child_samples: minimum records in a leaf (increases with dataset size to prevent overfitting) - max_bin: number of bins for feature bucketing (lower = faster but less precise)

LightGBM supports categorical features natively without one-hot encoding. Just specify the columns: model = lgb.LGBMClassifier() model.fit(X_train, y_train, categorical_feature=['City', 'Product'])

Feature importance: lgb.plot_importance(model, max_num_features=20)

LightGBM is often 5-10x faster than XGBoost on large datasets and achieves comparable or better accuracy. It is the algorithm of choice for large-scale tabular data problems.

Practical tip: On datasets under 10,000 rows, XGBoost or even Random Forest may outperform LightGBM because leaf-wise growth overfits small data. Use LightGBM when you have 50,000+ rows.

Exam tip: The num_leaves parameter controls model complexity more than max_depth does. A good rule: set num_leaves to roughly 2^max_depth but never more than about 70% of the number of training samples.