Attention versus contrastive learning of tabular data: a data-centric benchmarking

Despite groundbreaking success in image and text learning, deep learning has not achieved significant improvements against traditional machine learning (ML) on tabular data. This performance gap underscores the need for data-centric treatment and benchmarking of learning algorithms because tabular data with heterogeneous feature space differ in several ways from homogeneous image data. Recently, attention and contrastive learning breakthroughs have shifted computer vision and natural language processing paradigms. However, the effectiveness of these advanced deep models on tabular data is sparsely studied using a few data sets with very large sample sizes, reporting mixed findings after benchmarking against a limited number of baselines. We argue that the heterogeneity of tabular data sets and selective baselines in the literature can bias the benchmarking outcomes. This article extensively evaluates state-of-the-art attention and contrastive learning methods on 14 easy and 14 hard tabular data sets against traditional deep and machine learning. The proposed data-centric benchmarking demonstrates when traditional ML is preferred over deep learning and vice versa because no best learning method exists for all tabular data sets. Combining between-sample and between-feature attentions significantly outperforms state-of-the-art ML methods on tabular data. However, attention-based methods struggle with high-dimensional tabular data, where contrastive learning is more effective. While a hybrid attention-contrastive learning strategy mostly wins on hard data sets, traditional methods are frequently superior on easy data sets with presumably simpler decision boundaries. This is the first benchmarking paper with statistical comparisons of attention and contrastive learning performances on a diverse selection of tabular data sets against traditional deep and machine learning baselines to facilitate further advances in this field.