Awesome Gradient Boosting Machines

A comprehensive collection of gradient boosting libraries, cutting-edge research, practical tutorials, and industry resources—your one-stop guide to mastering GBMs.

From Kaggle competitions to production ML systems, gradient boosting machines dominate tabular data. This list tracks the ecosystem: battle-tested implementations, state-of-the-art papers from top venues (NeurIPS, ICML, ICLR), and resources to level up your GBM skills.

---

Contents

• Implementations
  • XGBoost
  • LightGBM
  • CatBoost
  • Other Frameworks
• Research Papers
  • 2025
  • 2024
  • 2023
  • 2022
  • 2021
  • 2020
  • 2019
  • Foundational Papers
• Tutorials & Guides
• Interpretability & Explainability
• Blog Posts
• Videos & Talks
• Books
• Benchmarks & Comparisons
• Related Awesome Lists
• Workshops
• 2026 Conferences
• Contributing

---

Implementations

XGBoost

• XGBoost - Scalable, portable, and distributed gradient boosting library. ⭐ 26k+
  • v2.0+ (2023): Improved memory usage, vector-leaf tree methods, and PySpark support.
• XGBoost Documentation - Official documentation with tutorials and API reference.
• XGBoost Python Package - Python wrapper for XGBoost.
• XGBoost4J-Spark - XGBoost integration with Apache Spark.

LightGBM

• LightGBM - A fast, distributed, high-performance gradient boosting framework by Microsoft. ⭐ 17k+
  • v4.0+ (2023): Quantized training support, improved GPU performance, and new objectives.
• LightGBM Documentation - Official documentation and tutorials.
• LightGBM Python Package - Python wrapper for LightGBM.

CatBoost

• CatBoost - A fast, scalable, high-performance gradient boosting on decision trees library by Yandex. ⭐ 8k+
• CatBoost Documentation - Official documentation with tutorials.
• CatBoost Python Package - Python wrapper for CatBoost.
• CatBoost Tutorials - Official tutorials and examples.

Other Frameworks

• NGBoost - Natural gradient boosting for probabilistic prediction by Stanford ML Group.
• XGBoost-Distribution - Probabilistic prediction with XGBoost via MLE. Like NGBoost but ~15x faster, with full XGBoost features (monotonic constraints, GPU). ⭐ 120+
• XGBoostLSS - Distributional gradient boosting extending XGBoost to model all parameters of 100+ distributions (location, scale, shape). Supports normalizing flows, mixture densities, and zero-inflated distributions. [Paper]
• LightGBMLSS - Distributional gradient boosting extending LightGBM. Model full conditional distributions with 50+ univariate distributions, normalizing flows, and mixture densities.
• gamboostLSS - R package for boosting GAMLSS (Generalized Additive Models for Location, Scale, Shape). High-dimensional distributional regression with gradient boosting. [JSS Paper] ⭐ 70+
• PGBM - Probabilistic Gradient Boosting Machines with native GPU acceleration, auto-differentiation, and uncertainty estimates. Built on PyTorch/Numba.
• GPBoost - Combines gradient boosting with Gaussian process and mixed effects models for spatial/grouped data. Handles random effects and spatial correlations. [JMLR Paper] ⭐ 600+
• GBNet - Integrates XGBoost/LightGBM with PyTorch for auto-differentiation of custom loss functions and hybrid neural network + GBM models. [JOSS Paper]
• Perpetual - Hyperparameter-free gradient boosting that self-generalizes. Just set a budget parameter instead of tuning hyperparameters. Written in Rust with Python bindings.
• GBDT-PL - Gradient Boosting with Piece-Wise Linear Regression Trees. Accelerates convergence and optimized for SIMD parallelism. (Now available in LightGBM via linear_tree=true)
• SGTB - Structured Gradient Tree Boosting for collective entity disambiguation by Bloomberg.
• AugBoost - Gradient boosting with step-wise feature augmentation using neural networks. Creates new features during boosting iterations. [IJCAI 2019 Paper]
• ThunderGBM - GPU-accelerated gradient boosting decision tree library.
• GrowNet - Gradient boosting with shallow neural networks as weak learners. Corrective step + dynamic learning rate. [Paper]
• GRANDE - End-to-end gradient-based training of decision tree ensembles; differentiable soft trees. [Paper]
• InterpretML / EBM - Microsoft's Explainable Boosting Machine - a glass-box model as accurate as black-box GBMs but fully interpretable. ⭐ 6k+
• FLAML - Microsoft's Fast and Lightweight AutoML library with efficient GBM hyperparameter tuning. ⭐ 4k+
• AutoGluon-Tabular - Amazon's AutoML that ensembles multiple GBMs (XGBoost, LightGBM, CatBoost) for state-of-the-art tabular performance. ⭐ 8k+
• AutoXGB - XGBoost + Optuna: auto train, tune, and serve XGBoost models directly from CSV. By Kaggle Grandmaster Abhishek Thakur. ⭐ 725+
• Scikit-learn HistGradientBoosting - Fast histogram-based GBM inspired by LightGBM, native to scikit-learn.
• Scikit-learn GradientBoosting - Classic gradient boosting implementation in scikit-learn.
• H2O GBM - H2O's gradient boosting machine implementation.
• TensorFlow Decision Forests - Decision forest algorithms including gradient boosted trees in TensorFlow.
• RAPIDS cuML - GPU machine learning library including XGBoost/gradient boosting support.
• SnapML - IBM's library for training generalized linear models and gradient boosting machines.
• PyGBM - Experimental gradient boosting machines in Python (archived).

---

Research Papers

2025

• Gradient Boosting Reinforcement Learning (ICML 2025)

  • Fuhrer, B., Tessler, C., & Dalal, G.
  • [Paper]
  • Adapts GBT to RL tasks, outperforming neural networks on structured/categorical data with better OOD robustness.

• Robust-Multi-Task Gradient Boosting (ICML 2025)

  • Emami, S., Martínez-Muñoz, G., & Hernández-Lobato, D.
  • [Paper]
  • R-MTGB framework that models task heterogeneity, detects outlier tasks, and promotes knowledge transfer.

• Gradient Boosted Mixed Models (ICML 2025)

  • Prevett, M. L., Hui, F. K. C., Tho, Z. Y., Welsh, A. H., & Westveld, A. H.
  • [Paper]
  • GBMixed extends boosting to jointly estimate mean and variance components for clustered data.

• Inductive Inference of Gradient-Boosted Decision Trees on Graphs (ICML 2025)

  • Vandervorst, F., Deprez, B., Verbeke, W., & Verdonck, T.
  • [Paper]
  • Graph gradient boosting machine (G-GBM) for supervised learning on heterogeneous/dynamic graphs.

• Quadratic Upper Bound for Boosting Robustness (ICML 2025)

  • You, E., & Lee, H. W.
  • [Paper]
  • Theoretical insights into stability and performance of gradient boosting algorithms.

• Statistical Inference for Gradient Boosting Regression (NeurIPS 2025)

  • [Paper]
  • Unified framework for statistical inference with dropout and parallel training; establishes central limit theorem for boosting.

• MorphBoost: Self-Organizing Universal Gradient Boosting with Adaptive Tree Morphing (NeurIPS 2025)

  • [Paper]
  • Self-organizing tree structures that dynamically adjust splitting behavior during training.

• GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models (NeurIPS 2025)

  • [Paper]
  • Uses pre-trained tabular models as surrogate with gradient info for high-dimensional optimization.

• NRGBOOST: Energy-Based Generative Boosted Trees (ICLR 2025)

  • Bravo, J.
  • [Paper]
  • Energy-based generative boosting analogous to XGBoost's second-order boosting; enables generative modeling on tabular data.

• Decision Trees That Remember: Recurrent Decision Trees with Memory (ICLR 2025)

  • Marton, S., Schneider, M., Brinkmann, J., Lüdtke, S., Bartelt, C., & Stuckenschmidt, H.
  • [Paper]
  • ReMeDe Trees: recurrent decision trees with memory for sequential data and long-term dependencies.

• Boosting Methods for Interval-Censored Data (ICLR 2025)

  • Bian, Y., Yi, G. Y., & He, W.
  • [Paper] [Code]
  • Nonparametric boosting for regression/classification with interval-censored data; L2Boost-CUT and L2Boost-IMP algorithms.

• Online Gradient Boosting Decision Tree: In-Place Updates for Adding/Deleting Data (ICLR 2025)

  • [Paper]
  • Online GBDT with incremental and decremental learning support.

• GAdaBoost: Efficient and Robust AdaBoost Based on Granular-Ball Structure (AAAI 2025)

  • Xie, Q., et al.
  • [Paper]
  • Two-stage framework using granular-ball data compression; robust under noisy conditions for multiclass classification.

• Representation Learning for Tabular Data: A Comprehensive Survey (arXiv 2025)

  • Jiang, J. P., Liu, S. Y., Cai, H. R., Zhou, Q. L., & Ye, H. J.
  • [Paper] [Code]
  • Comprehensive survey covering specialized, transferable, and foundation models for tabular data; compares DNNs vs GBDTs with taxonomy of representation learning strategies.

• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling (ICLR 2025)

  • [Paper] [Code]
  • Simple MLP-based architecture with parameter-efficient ensembling; competitive with GBDTs while being much simpler than other deep tabular methods.

• TabPFN v2: Accurate Predictions on Small Data with a Tabular Foundation Model (Nature 2025)

  • [Paper] [Code]
  • Tabular foundation model achieving state-of-the-art on small datasets; extends TabPFN with improved scaling and ensembling.

• TabArena: A Living Benchmark for Machine Learning on Tabular Data (NeurIPS 2025)

  • [Paper] [Code]
  • Living benchmark with continuous updates; tracks performance of GBDTs vs deep learning on tabular tasks.

• TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks (ICLR 2025)

  • [Paper] [Code]
  • Identifies critical pitfalls in tabular benchmarks; introduces new datasets with temporal train-test splits where GBDTs maintain advantages.

• LLM-Boost & PFN-Boost: Transformers Boost the Performance of Decision Trees on Tabular Data (arXiv 2025)

  • [Paper] [Code]
  • Hybrid ensembles combining TabPFN/LLMs with decision trees; transformers boost tree performance across sample sizes.

• Beta: TabPFN Unleashed - A Scalable and Effective Solution to Tabular Classification (arXiv 2025)

  • [Paper]
  • Scales TabPFN to larger datasets; maintains competitive performance with GBDTs on bigger tabular problems.

• GReaTER: Generate Realistic Tabular Data after Data Enhancement and Reduction (ICDE 2025 Workshop)

  • Kwok, T. S. T., Wang, C. H., & Cheng, G.
  • [Paper]
  • Extends GReaT framework with semantic enhancement and cross-table connections for multi-table synthesis using LLMs.

2024

• A Closer Look at Deep Learning Methods on Tabular Datasets (arXiv 2024)

  • Ye, H. J., Liu, S. Y., Cai, H. R., Zhou, Q. L., & Zhan, D. C.
  • [Paper] [Code]
  • TALENT benchmark with 300+ datasets; shows GBDTs remain strong baselines while pretrained tabular models are closing the gap. Introduces TALENT-tiny (45 datasets) for rapid evaluation.

• Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data (NeurIPS 2024)

  • [Paper] [Code]
  • Proposes well-tuned default hyperparameters for GBDTs and MLPs; shows proper tuning closes gap between methods.

• GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Data (ICLR 2024)

  • [Paper] [Code]
  • End-to-end gradient-based training of decision tree ensembles; combines differentiability of neural nets with interpretability of trees.

• HyperFast: Instant Classification for Tabular Data (AAAI 2024)

  • [Paper] [Code]
  • Meta-learned hypernetwork for instant tabular classification; no training needed on new datasets.

• A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data (NeurIPS 2024)

  • [Paper] [Code]
  • Data-centric benchmark examining how dataset properties affect GBDT vs DNN performance; provides guidelines for model selection.

• GTL: From Supervised to Generative - A Novel Paradigm for Tabular Deep Learning with LLMs (KDD 2024)

  • [Paper] [Code]
  • Microsoft's generative approach to tabular learning; LLM-based predictions competitive with GBDTs.

• Optimal Parallelization of Boosting (arXiv 2024)

  • da Cunha, A., Høgsgaard, M. M., & Larsen, K. G.
  • [Paper]
  • Establishes tight lower bounds on the parallel complexity of boosting; presents algorithm matching bounds across the full rounds vs. work spectrum.

• TabR: Tabular Deep Learning Meets Nearest Neighbors (ICLR 2024)

  • Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., & Babenko, A.
  • [Paper] [Code]
  • Retrieval-augmented deep learning for tabular data; competitive with tuned GBMs.

• Language Models are Realistic Tabular Data Generators (ICLR 2024)

  • Borisov, V., Seßler, K., Leemann, T., Heyer, M., & Kasneci, G.
  • [Paper] [Code]
  • GReaT: Generating Realistic Tabular data with Large Language Models.

• T2G-Former: Organizing Tabular Features via Relation-Weighted Graph for Tree-Based Models (AAAI 2024)

  • Yan, J., et al.
  • [Paper]
  • Graph-based feature organization to enhance tree-based models.

• MSBoost: Using Model Selection with Multiple Base Estimators for Gradient Boosting (Preprint 2024)

  • [Paper]
  • Heterogeneous base learners: trains multiple model types in parallel per boosting round, selects best. Strong on small/noisy datasets.

2023

• When Do Neural Nets Outperform Boosted Trees on Tabular Data? (NeurIPS 2023)

  • McElfresh, D., et al.
  • [Paper] [Code]
  • TabZilla meta-learning analysis on 36 datasets; identifies when neural nets beat GBDTs based on dataset characteristics.

• Orthogonal Gradient Boosting for Interpretable Additive Rule Ensembles (NeurIPS 2023)

  • Yang, F., Le Bodic, P., Kamp, M., & Boley, M.
  • [Paper]
  • Orthogonal boosting for rule ensembles; produces interpretable additive models while maintaining predictive performance.

• TableShift: A Benchmark for Distribution Shift in Tabular Data (NeurIPS 2023)

  • Gardner, J., et al.
  • [Paper] [Code]
  • Benchmark for evaluating robustness to distribution shift; tests GBDTs vs deep learning under domain shift.

• A Benchmark of Categorical Encoders for Binary Classification (NeurIPS 2023)

  • Cerda, P., & Varoquaux, G.
  • [Paper] [Code]
  • Comprehensive benchmark of categorical encoders; guidance for feature engineering with GBDTs.

• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (ICLR 2023)

  • Hollmann, N., Müller, S., Eggensperger, K., & Hutter, F.
  • [Paper] [Code]
  • Prior-data fitted network for small tabular tasks; no training needed, competitive with tuned GBMs.

• TabDDPM: Modelling Tabular Data with Diffusion Models (ICML 2023)

  • Kotelnikov, A., Barber, D., & Babenko, A.
  • [Paper] [Code]
  • Diffusion models for tabular data generation; state-of-the-art synthetic data quality.

• XTab: Cross-table Pretraining for Tabular Transformers (ICML 2023)

  • Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G., & Shoaran, M.
  • [Paper] [Code]
  • Federated pretraining across multiple tables; transferable representations for tabular data.

2022

• MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees (arXiv 2022)

  • Ying, Z., Xu, Z., Li, Z., Wang, W., & Meng, C.
  • [Paper]
  • GBDT-based multi-task learning with shared tree structures; assigns multiple outputs per leaf and combines gradients across tasks.

• GWRBoost: A Geographically Weighted Gradient Boosting Method for Explainable Quantification of Spatially-Varying Relationships (arXiv 2022)

  • Wang, H., Huang, Z., Yin, G., Bao, Y., Zhou, X., & Gao, Y.
  • [Paper]
  • Combines GWR with gradient boosting for spatial heterogeneity; improves parameter estimation by 18.3% and reduces AICc by 67.3% vs. classic GWR.

• AGBoost: Attention-based Modification of Gradient Boosting Machine (arXiv 2022)

  • Konstantinov, A., Utkin, L., & Kirpichenko, S.
  • [Paper]
  • Assigns trainable attention weights to GBM iterations; learns to weight contribution of each boosting round.

• Elastic Gradient Boosting Decision Tree with Adaptive Iterations for Concept Drift Adaptation (TKDE 2022)

  • Wang, K., Lu, J., Liu, A., Song, Y., Xiong, L., & Zhang, G.
  • [Paper]
  • Elastic GBDT that adapts to concept drift by adjusting iterations; robust in dynamic/streaming environments.

• Quantized Training of Gradient Boosting Decision Trees (NeurIPS 2022)

  • Shi, Y., Ke, G., Chen, Z., Zheng, S., & Liu, T. Y.
  • [Paper]
  • Low-precision (2-3 bit) gradient quantization for GBDT; up to 2x speedup. Now in LightGBM.

• Why do tree-based models still outperform deep learning on tabular data? (NeurIPS 2022)

  • Grinsztajn, L., Oyallon, E., & Varoquaux, G.
  • [Paper] [Code]
  • Landmark benchmark: GBDTs beat DNNs on medium-sized tabular data due to irregular patterns and uninformative features.

• Gradient Boosted Decision Tree Neural Network (ICLR 2022)

  • Chen, S., Guestrin, C.
  • [Paper]
  • Hybrid architecture combining GBDT structure with neural network training.

• NODE: Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (ICLR 2020)

  • Popov, S., Morozov, S., & Babenko, A.
  • [Paper] [Code]
  • Differentiable oblivious decision trees (like CatBoost) trained end-to-end; competitive with GBMs.

• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (NeurIPS 2022)

  • Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T.
  • [Paper] [Code]
  • Self-attention over both rows and columns; contrastive pre-training for tabular data.

2021

• Challenges and Opportunities of Building Fast GBDT Systems (IJCAI 2021 Survey)

  • Wen, Z., Li, Q., He, B., & Cui, B.
  • [Paper]
  • Comprehensive survey on parallel/distributed GBDT: multi-core CPUs, GPUs, clusters, federated learning, and privacy-preserving GBDTs.

• Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression (KDD 2021)

  • Sprangers, O., Schelter, S., & de Rijke, M.
  • [Paper] [Code]

• TabNet: Attentive Interpretable Tabular Learning (AAAI 2021)

  • Arik, S. Ö., & Pfister, T.
  • [Paper] [Code]
  • Sparse attention for feature selection; interpretable DNN competitive with GBMs on some tasks.

• Revisiting Deep Learning Models for Tabular Data (NeurIPS 2021)

  • Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A.
  • [Paper] [Code]
  • Introduces FT-Transformer; comprehensive benchmark comparing ResNet, MLP, Transformer architectures to GBMs on tabular data.

• Tabular Data: Deep Learning is Not All You Need (arXiv 2021)

  • Shwartz-Ziv, R., & Armon, A.
  • [Paper]
  • Benchmarks XGBoost against deep models on 11 datasets; XGBoost outperforms or matches DNNs, with much faster training and easier tuning.

• Regularization is All You Need: Simple Neural Nets Can Excel on Tabular Data (NeurIPS 2021)

  • Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J.
  • [Paper]
  • Cocktail of regularization techniques makes simple MLPs competitive with GBMs.

• DP-XGBoost: Private Machine Learning at Scale (arXiv 2021)

  • Grammenos, A., Mendoza Smith, R., Crowcroft, J., & Mascolo, C.
  • [Paper]
  • Extends XGBoost with differential privacy mechanisms for large-scale privacy-preserving training.

• Fed-EINI: An Efficient and Interpretable Inference Framework for Decision Tree Ensembles in Vertical Federated Learning (BigData 2021)

  • Chen, X., Zhou, S., Yang, K., et al.
  • [Paper]
  • Efficient and interpretable inference for tree ensembles in vertical federated learning scenarios.

• Net-DNF: Effective Deep Modeling of Tabular Data (ICLR 2021)

  • Katzir, L., Elidan, G., & El-Yaniv, R.
  • [Paper]
  • Neural architecture with inductive bias for DNF-like Boolean formulas over soft-threshold decision terms; promotes localized decisions on feature subsets, outperforms fully connected networks on tabular data.

2020

• From Local Explanations to Global Understanding with Explainable AI for Trees (Nature Machine Intelligence 2020)

  • Lundberg, S. M., et al.
  • [Paper] [Code]
  • TreeSHAP: polynomial-time exact SHAP for trees. Global feature importance, dependence plots, and interaction detection.

• Survival Regression with Accelerated Failure Time Model in XGBoost (JCGS 2022, arXiv 2020)

  • Barnwal, A., Cho, H., & Hocking, T.
  • [Paper] [Code]
  • First GPU-accelerated AFT model; supports right/left/interval censoring for survival analysis. Now in XGBoost.

• NGBoost: Natural Gradient Boosting for Probabilistic Prediction (ICML 2020)

  • Duan, T., Avati, A., Ding, D. Y., Basu, S., Ng, A. Y., & Schuler, A.
  • [Paper] [Code]
  • Natural gradient for probabilistic prediction; outputs full distributions, not just point estimates.

• GrowNet: Gradient Boosting Neural Networks (arXiv 2020)

  • Badirli, S., Liu, X., Xing, Z., Bhowmik, A., Doan, K., & Keerthi, S. S.
  • [Paper] [Code]
  • Gradient boosting with shallow neural networks as weak learners; combines boosting with representation learning.

• TabTransformer: Tabular Data Modeling Using Contextual Embeddings (arXiv 2020)

  • Huang, X., Khetan, A., Cvitkovic, M., & Karnin, Z.
  • [Paper] [Code]
  • Transformer for categorical features with contextual embeddings; foundational work comparing transformers to GBDTs on tabular data.

• Privacy-Preserving Gradient Boosting Decision Trees (AAAI 2020)

  • Li, Q., Wu, Z., Wen, Z., & He, B.
  • [Paper]
  • Differential privacy for GBDT training; tighter sensitivity bounds and optimized noise allocation for better accuracy-privacy tradeoff.

• Learning Gradient Boosted Multi-label Classification Rules (ECML-PKDD 2020)

  • Rapp, M., Loza Mencía, E., Fürnkranz, J., Nguyen, V. L., & Hüllermeier, E.
  • [Paper] [Code]
  • BOOMER: interpretable gradient boosted rules for multi-label classification; balances accuracy and interpretability.

• Gradient Boosting Machine with Partially Randomized Decision Trees (CoRR 2020)

  • Konstantinov, A. V., & Utkin, L. V.
  • [Paper]
  • Uses partially randomized trees (à la extremely randomized trees) in gradient boosting; reduces discontinuity issues and computational complexity.

• Soft Gradient Boosting Machine (CoRR 2020)

  • Feng, J., Xu, Y. X., Jiang, Y., & Zhou, Z. H.
  • [Paper]
  • Differentiable soft decision trees with joint optimization; enables parallel training and online/incremental learning for GBMs.

• StructureBoost: Efficient Gradient Boosting for Structured Categorical Variables (CoRR 2020)

  • Lucena, B.
  • [Paper]
  • Handles categorical variables with known hierarchical/graph structure; outperforms CatBoost/LightGBM on structured categoricals and generalizes to unseen categories.

• Gradient Boosted Decision Trees for High Dimensional Sparse Output (ICML 2020)

  • Si, S., Zhang, H., Keerthi, S. S., Mahajan, D., Dhillon, I. S., & Hsieh, C. J.
  • [Paper]

• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (ICML 2020 AutoML Workshop)

  • Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., & Smola, A.
  • [Paper] [Code]

2019

• Regularized Gradient Boosting (NeurIPS 2019)

  • Cortes, C., & Mohri, M.
  • [Paper]
  • RGB algorithm: applies Structural Risk Minimization at each boosting round to balance empirical fit and model complexity; provides data-dependent generalization guarantees via Rademacher complexity.

• Gradient Boosting with Piece-Wise Linear Regression Trees (IJCAI 2019)

  • Shi, Y., Li, J., & Li, Z.
  • [Paper] [Code]
  • Linear models in leaves instead of constants; faster convergence with fewer trees. Now in LightGBM.

• AugBoost: Gradient Boosting Enhanced with Step-Wise Feature Augmentation (IJCAI 2019)

  • Tannor, P., & Rokach, L.
  • [Paper] [Code]
  • Uses neural networks to create new features at each boosting iteration; augments feature space progressively during training.

• CatBoost: Unbiased Boosting with Categorical Features (NeurIPS 2018)

  • Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A.
  • [Paper] [Code]
  • Ordered boosting prevents target leakage; native categorical encoding without one-hot.

• gbdt-rs: Fast and Trustworthy Gradient Boosting Decision Tree (IEEE S&P 2019 Poster)

  • Li, T., Li, T., Ding, Y., Zhang, Y., Wei, T., & Han, X.
  • [Paper]
  • Rust implementation for Intel SGX secure enclaves; 10x faster inference than XGBoost, memory-safe, ~2000 LOC.

• SecureBoost: A Lossless Federated Learning Framework (IEEE Intelligent Systems 2019)

  • Cheng, K., Fan, T., Jin, Y., Liu, Y., Chen, T., & Yang, Q.
  • [Paper]
  • Vertical federated learning for XGBoost; enables secure collaborative training across data silos without sharing raw data.

• SecureGBM: Secure Multi-Party Gradient Boosting (IEEE BigData 2019)

  • Feng, Z., Xiong, H., Song, C., et al.
  • [Paper]
  • Secure multi-party computation framework for GBMs; enables collaborative training while preserving data privacy.

2018

• Collective Entity Disambiguation with Structured Gradient Tree Boosting (NAACL 2018)

  • Yang, Y., Irsoy, O., & Rahman, K. S.
  • [Paper] [Code]
  • Structured prediction with GBDTs for entity linking; models entity interdependencies.

• Multi-Layered Gradient Boosting Decision Trees (NeurIPS 2018)

  • Feng, J., Yu, Y., & Zhou, Z. H.
  • [Paper] [Code]
  • Multi-layer representation learning with GBDTs; deep forest alternative to deep neural nets.

• Learning Deep ResNet Blocks Sequentially using Boosting Theory (ICML 2018)

  • Huang, F., Ash, J. T., Langford, J., & Schapire, R. E.
  • [Paper] [Code]
  • Boosting perspective on ResNets; proves ResNet blocks can be learned sequentially.

• Functional Gradient Boosting based on Residual Network Perception (ICML 2018)

  • Nitanda, A., & Suzuki, T.
  • [Paper] [Code]
  • Functional gradient descent with neural network base learners.

• Finding Influential Training Samples for Gradient Boosted Decision Trees (ICML 2018)

  • Sharchilev, B., Ustinovskiy, Y., Serdyukov, P., & de Rijke, M.
  • [Paper]
  • LeafInfluence/LeafRefit: efficiently compute influence of training samples on GBDT predictions.

• Boosting Variational Inference: an Optimization Perspective (AISTATS 2018)

  • Locatello, F., Khanna, R., Ghosh, J., & Rätsch, G.
  • [Paper] [Code]
  • Applies boosting to variational inference; iteratively adds mixture components.

2017

• A Unified Approach to Interpreting Model Predictions (NeurIPS 2017)

  • Lundberg, S. M., & Lee, S. I.
  • [Paper] [Code]
  • Introduces SHAP values unifying LIME, DeepLIFT, and Shapley values. The gold standard for explaining GBM predictions. 30k+ citations.

• LightGBM: A Highly Efficient Gradient Boosting Decision Tree (NeurIPS 2017)

  • Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.
  • [Paper] [Code]
  • GOSS (gradient-based one-side sampling) + EFB (exclusive feature bundling) for 20x speedup over XGBoost.

• AdaGAN: Boosting Generative Models (NeurIPS 2017)

  • Tolstikhin, I. O., Gelly, S., Bousquet, O., Simon-Gabriel, C. J., & Schölkopf, B.
  • [Paper] [Code]
  • Boosting framework for GANs; iteratively trains generators to cover missing modes.

• Gradient Boosted Decision Trees for High Dimensional Sparse Output (ICML 2017)

  • Si, S., Zhang, H., Keerthi, S. S., Mahajan, D., Dhillon, I. S., & Hsieh, C. J.
  • [Paper] [Code]
  • Efficient GBDT for extreme multi-label classification with millions of labels.

• Variational Boosting: Iteratively Refining Posterior Approximations (ICML 2017)

  • Miller, A. C., Foti, N. J., & Adams, R. P.
  • [Paper] [Code]
  • Gradient boosting for Bayesian inference; refines variational approximations.

• BDT: Gradient Boosted Decision Tables for High Accuracy and Scoring Efficiency (KDD 2017)

  • Lou, Y., & Obukhov, M.
  • [Paper]
  • Decision tables as base learners; 10x faster scoring than trees with comparable accuracy.

• Gradient Boosting on Stochastic Data Streams (AISTATS 2017)

  • Hu, H., Sun, W., Venkatraman, A., Hebert, M., & Bagnell, J. A.
  • [Paper]
  • Online boosting for streaming data; adapts to non-stationary distributions.

2016

• XGBoost: A Scalable Tree Boosting System (KDD 2016)

  • Chen, T., & Guestrin, C.
  • [Paper] [Code]
  • The foundational XGBoost paper; sparsity-aware splits, weighted quantile sketch, cache-aware access. 50k+ citations.

• Boosting with Abstention (NeurIPS 2016)

  • Cortes, C., DeSalvo, G., & Mohri, M.
  • [Paper]
  • Boosting with option to abstain from prediction; theoretical analysis of abstention costs.

• Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition (NeurIPS 2016)

  • Han, S., Meng, Z., Khan, A. S., & Tong, Y.
  • [Paper] [Code]
  • Boosted ensemble of CNNs for facial action unit detection.

• Boosted Decision Tree Regression Adjustment for Variance Reduction (KDD 2016)

  • Poyarkov, A., Drutsa, A., Khalyavin, A., Gusev, G., & Serdyukov, P.
  • [Paper]
  • GBDT for variance reduction in A/B testing; improves statistical power of online experiments.

• L-EnsNMF: Boosted Local Topic Discovery via Ensemble of NMF (ICDM 2016)

  • Suh, S., Choo, J., Lee, J., & Reddy, C. K.
  • [Paper] [Code]
  • Boosting for matrix factorization; ensemble of NMF for topic modeling.

2015

• DART: Dropouts meet Multiple Additive Regression Trees (AISTATS 2015)

  • Rashmi, K. V., & Gilad-Bachrach, R.
  • [Paper]
  • Dropout regularization for boosted trees; prevents over-specialization of later trees. Now available in XGBoost and LightGBM.

• Online Gradient Boosting (NeurIPS 2015)

  • Beygelzimer, A., Hazan, E., Kale, S., & Luo, H.
  • [Paper] [Code]

• Efficient Second-Order Gradient Boosting for Conditional Random Fields (AISTATS 2015)

  • Chen, T., Singh, S., Taskar, B., & Guestrin, C.
  • [Paper]

• Optimal Action Extraction for Random Forests and Boosted Trees (KDD 2015)

  • Cui, Z., Chen, W., He, Y., & Chen, Y.
  • [Paper]

• A Boosting Algorithm for Item Recommendation with Implicit Feedback (IJCAI 2015)

  • Liu, Y., Zhao, P., Sun, A., & Miao, C.
  • [Paper]

Foundational Papers

• Greedy Function Approximation: A Gradient Boosting Machine (Annals of Statistics 2001)

  • Friedman, J. H.
  • [Paper]

• Stochastic Gradient Boosting (Computational Statistics & Data Analysis 2002)

  • Friedman, J. H.
  • [Paper]

• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting (JCSS 1997)

  • Freund, Y., & Schapire, R. E.
  • [Paper]

• Experiments with a New Boosting Algorithm (ICML 1996)

  • Freund, Y., & Schapire, R. E.
  • [Paper]

• Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods (Annals of Statistics 1998)

  • Schapire, R. E., Freund, Y., Bartlett, P., & Lee, W. S.
  • [Paper]

---

Tutorials & Guides

Getting Started

• Introduction to Gradient Boosting - Machine Learning Mastery
• A Gentle Introduction to XGBoost - Machine Learning Mastery
• Complete Guide to LightGBM - Official Quick Start
• CatBoost Tutorial - Official Tutorial

Parameter Tuning

• XGBoost Parameters Guide - Official Parameters Documentation
• LightGBM Parameters Tuning - Official Tuning Guide
• CatBoost Training Parameters - Official Parameters Reference
• Hyperparameter Tuning with Optuna - Automatic hyperparameter optimization framework

Advanced Topics

• Custom Loss Functions in XGBoost - Custom Objective and Evaluation
• Distributed Training with XGBoost - XGBoost with Dask
• GPU Training with LightGBM - GPU Acceleration Guide
• Handling Missing Values - CatBoost Missing Values

Interpretability & Explainability

• SHAP - SHapley Additive exPlanations for interpreting GBM predictions. The gold standard for feature importance. ⭐ 23k+
• PyCEbox - Individual Conditional Expectation (ICE) plots for peeking inside black-box models. Shows how predictions vary per feature for individual instances, complementing PDPs. [Paper] ⭐ 164+
• dtreeviz - Beautiful decision tree visualization and model interpretation. Supports XGBoost, LightGBM, sklearn, PySpark, TensorFlow DF. Now with AI-powered explanations! ⭐ 3.1k+
• ELI5 - Debug ML classifiers and explain predictions. Supports XGBoost, LightGBM, CatBoost, sklearn. Includes LIME for text and permutation importance. ⭐ 2.8k+
• SHAP TreeExplainer Tutorial - Efficient SHAP values for tree models
• InterpretML EBM Tutorial - Glass-box interpretable boosting
• Permutation Importance - Model-agnostic feature importance

---

Blog Posts

Comparisons & Benchmarks

• XGBoost vs LightGBM vs CatBoost - Comprehensive Comparison
• When to Use Different Gradient Boosting Libraries - Neptune.ai Blog
• Gradient Boosting vs Random Forest - Comparative Analysis

Deep Dives

• How Gradient Boosting Works - Visual and Mathematical Explanation
• Understanding LightGBM's Histogram-based Algorithm - Technical Deep Dive
• CatBoost: Handling Categorical Features - Feature Engineering with CatBoost
• XGBoost Mathematics Explained - Mathematical Foundation

Practical Applications

• Winning Kaggle Competitions with Gradient Boosting - Kaggle Tutorial
• Feature Engineering for Gradient Boosting - Real Competition Insights
• Time Series Forecasting with Gradient Boosting - Time Series Applications

---

Videos & Talks

Conference Talks

• XGBoost: A Scalable Tree Boosting System - Tianqi Chen, KDD 2016
• LightGBM: A Highly Efficient Gradient Boosting Decision Tree - Microsoft Research
• CatBoost - the new generation of gradient boosting - Yandex

Educational Videos

• Gradient Boosting Explained - StatQuest with Josh Starmer
• XGBoost Part 1: Regression - StatQuest
• XGBoost Part 2: Classification - StatQuest
• AdaBoost, Clearly Explained - StatQuest

Practical Tutorials

• XGBoost Python Tutorial - Hands-on Implementation
• LightGBM Complete Guide - Complete Tutorial
• CatBoost Tutorial for Beginners - Getting Started

---

Books

• Hands-On Gradient Boosting with XGBoost and scikit-learn - Corey Wade (Packt Publishing, 2020)

  • [Book]

• The Elements of Statistical Learning - Hastie, Tibshirani, & Friedman (Springer, 2009)

  • Chapter 10: Boosting and Additive Trees
  • [Free PDF]

• Pattern Recognition and Machine Learning - Christopher Bishop (Springer, 2006)

  • [Book]

• Machine Learning: A Probabilistic Perspective - Kevin Murphy (MIT Press, 2012)

  • [Book]

---

Benchmarks & Comparisons

Benchmark Repositories

• Tabular Benchmark - Comprehensive benchmark comparing deep learning and gradient boosting
• TabArena - Living benchmark with continuous updates for tabular ML methods
• LAMDA-TALENT - 300+ tabular datasets benchmark with TALENT-tiny for rapid evaluation
• TabZilla - Meta-learning analysis comparing neural nets vs boosted trees
• TabReD - Benchmark addressing pitfalls in tabular deep learning evaluation
• TableShift - Distribution shift benchmark for tabular data
• EncoderBenchmarking - Benchmark of categorical encoders for feature engineering
• pytabkit - Pre-tuned defaults for GBDTs and MLPs with benchmarking tools
• OpenML Benchmarking Suites - Standardized machine learning benchmarks
• MLPerf Training - Industry-standard ML benchmarks

Comparison Studies

• Deep Neural Networks vs Gradient Boosted Trees - Empirical comparison study
• AutoML Benchmark - Comparing automated ML systems
• When Do Neural Nets Outperform Boosted Trees? - TabZilla meta-learning analysis
• Data-Centric Tabular Evaluation - How dataset properties affect model selection

---

Related Awesome Lists

• Awesome Gradient Boosting Papers - Academic papers on gradient boosting
• Awesome Gradient Boosting - Resources for data scientists
• Awesome Tabular Deep Learning - Comprehensive survey on tabular representation learning (specialized, transferable, and foundation models)
• Awesome Tabular LLMs - Large language models for tabular data understanding and reasoning
• Awesome Decision Tree Research - Decision tree papers
• Awesome Machine Learning - General ML resources
• Awesome AutoML - AutoML papers and resources

---

Workshops

Specialized workshops on tabular data and tree-based methods:

• AI for Tabular Data @ EurIPS 2025 - Representation learning, generative AI, and foundation models for tables (Dec 6, 2025, Copenhagen)
• Table Representation Learning Workshop @ NeurIPS 2024 - Latest advances in tabular ML
• Table Representation Learning Workshop @ NeurIPS 2023
• Table Representation Learning Workshop @ NeurIPS 2022
• Table Representation Learning Workshop @ ACL 2025 - 4th TRL workshop (July 31, 2025, Vienna, Austria)
• Foundation Models for Structured Data @ ICML 2025 - TabPFN, TabICL, TabForestPFN for tabular; Chronos, TimesFM for time series (July 18, 2025, Vancouver)

---

2026 Conferences

Top ML/AI conferences in 2026 where gradient boosting research is commonly presented:

Note: Conference dates and locations are subject to change. Please check official websites for the most up-to-date information.

---

Contributing

Your contributions are always welcome! Please read the contribution guidelines first.

How to Contribute

1. Add a paper: Use the paper template or submit a PR
2. Add a resource: Use the resource template
3. Report broken links: Use the broken link template

If you have any questions about this list, don't hesitate to open an issue.

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

_{If you find this list useful, please consider giving it a ⭐️ star!}