Why Gradient-Boosted Trees Still Win on Tabular Data, and When They Do Not

Quick take. If your data lives in rows and columns, start with gradient-boosted trees. Three benchmark studies published in 2022 and 2023 found them competitive with or better than deep learning on typical tabular datasets. The exceptions are real but specific: small datasets, raw signals that are not really tabular, and cases where a small accuracy gain justifies an ensemble.

What counts as tabular

Tabular data is a table where each row is an example and each column is a feature with its own meaning and units. In operations, a row might be a production batch, and the columns might be furnace temperature, line speed, supplier code, hours since maintenance and a lab result. The columns are heterogeneous: numeric and categorical, often skewed, often with missing values.

Images, audio, vibration waveforms and free text are not tabular in this sense, even when they sit in a database.

I build predictive systems for enterprise operations, and many of those problems are tabular: rows of process readings, batches and outcomes. For data shaped like this, a gradient-boosted tree is a strong first model. The published evidence explains why.

How gradient boosting works

Friedman framed boosting as gradient descent in function space: instead of adjusting weights, you add functions, usually small trees, and each one moves the prediction downhill on the loss [1]. For squared error the recipe is short:

1. Start with a constant prediction F0, the mean of the target.
2. Compute the residual for each row: r = y - F(x). For squared error, the residual is the negative gradient.
3. Fit a small regression tree to the residuals.
4. Update the model: F(x) = F(x) + learning rate × tree(x).
5. Repeat steps 2-4, and stop when error on a validation set stops improving.

A toy example, with illustrative numbers: three batches with yields of 90, 94 and 98%.

• Round 0: F0 = 94 for every batch. Residuals: -4, 0 and +4.
• Round 1: a one-split tree on furnace temperature puts batch 1 alone (leaf value -4) and batches 2 and 3 together (leaf value 2, the mean of 0 and 4).
• With a learning rate of 0.1, predictions become 93.6, 94.2 and 94.2. Residuals shrink to -3.6, -0.2 and +3.8.

Each round removes a little of what earlier rounds got wrong. Hundreds of small trees add up to a flexible model built from simple threshold rules.

XGBoost, described by Chen and Guestrin in 2016, added a sparsity-aware split-finding algorithm that handles missing values, a weighted quantile sketch for approximate split finding, and systems engineering that lets boosting scale to very large datasets [2].

What the benchmarks say

Two details make these results hard to dismiss. Grinsztajn and colleagues spent about 20,000 compute hours on random hyperparameter search, roughly 400 iterations per dataset, so the deep models were not under-tuned [4]. Shwartz-Ziv and Armon tested TabNet, NODE, DNF-Net and a one-dimensional convolutional network, and XGBoost generally came out ahead, including on datasets used in the deep models’ own papers [3].

Why trees suit tabular data

Grinsztajn and colleagues traced the gap to inductive biases, the built-in assumptions each model family makes about the function it learns [4]:

• Smoothness. Neural networks are biased toward overly smooth solutions. Tabular targets often jump: a process runs fine until a temperature crosses a limit. A tree represents that jump with one split.
• Uninformative features. Irrelevant columns hurt plain feed-forward networks (multilayer perceptrons) more than trees. Operational tables usually carry many weak or irrelevant columns.
• Orientation. Tabular data is not invariant to rotation. Each column has its own meaning, and a dense layer that mixes columns from the first step blurs that structure. A tree splits on one column at a time.

McElfresh and colleagues add that boosted trees handle skewed or heavy-tailed feature distributions, and other dataset irregularities, much better than neural networks [5].

There are practical reasons too. Trees train quickly on ordinary processors and need no feature scaling, and common boosting libraries handle missing values natively, which keeps the production pipeline simple.

When trees do not win

• Small datasets. TabPFN, a transformer pretrained on more than 100 million synthetic tabular tasks, was reported in Nature in January 2025 to outperform previous methods by a wide margin on datasets of up to 10,000 samples [6]. In one classification setting it beat an ensemble of the strongest baselines tuned for four hours, in 2.8 seconds [6]. An earlier version was already the best algorithm on average in the McElfresh benchmark, even though it was effectively limited to 3,000 training rows [5].
• Data that is not really tabular. Weld images, vibration waveforms, spectra and maintenance notes need learned features. A practical pattern is to let a neural network turn the raw signal into a fixed-length embedding, then feed that embedding to a tree model alongside the process variables.
• Last-mile accuracy. An ensemble of deep models and XGBoost beat XGBoost alone in the Shwartz-Ziv and Armon study [3]. Weigh that gain against the cost of running and monitoring two model families.
• Joint training. If the tabular model must be trained end to end with an image or text model, a differentiable network is the natural choice.

A selection protocol

The 10,000-row and 500-feature limits in the second row come from the range where TabPFN’s strongest results were reported [6]. Then work through the comparison in order:

1. Fit a simple baseline, such as linear or logistic regression. It tells you how much the complex model really adds.
2. Train a boosted tree with library defaults and early stopping, then do a light tuning pass on learning rate, tree depth and number of rounds.
3. If the data is small, add a tabular foundation model to the comparison.
4. Try a deep tabular model last, on the same data splits.
5. For process data, split by time rather than at random, so the test set looks like the future and not like a shuffled past.
6. Choose on the decision metric and on the cost of running the model, not on validation score alone.

The benchmarks do not say deep learning is useless for tables. They say the burden of proof sits with the more complex model. Make it earn its place on your data, with your metric.

References

1. J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” The Annals of Statistics, vol. 29, no. 5, pp. 1189-1232, 2001.
2. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), 2016.
3. R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” Information Fusion, vol. 81, pp. 84-90, 2022.
4. L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), Datasets and Benchmarks Track, 2022.
5. D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White, “When do neural nets outperform boosted trees on tabular data?,” in Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, 2023.
6. N. Hollmann et al., “Accurate predictions on small data with a tabular foundation model,” Nature, vol. 637, pp. 319-326, 2025.