Awesome Self-Supervised Learning for Non-Sequential Tabular Data (SSL4NSTD)

This repository contains the frontier research on self-supervised learning for tabular data which has been a popular topic recently.
This list is maintained by Wei-Wei Du and Wei-Yao Wang. (Actively keep updating)
If you have come across relevant resources or found some errors in this repository, feel free to open an issue or submit a PR.

Our Survey Paper

A Survey on Self-Supervised Learning for Non-Sequential Tabular Data (ACML-24 Journal Track)

Citation

@article{DBLP:journals/corr/abs-2402-01204,
  author       = {Wei{-}Yao Wang and
                  Wei{-}Wei Du and
                  Derek Xu and
                  Wei Wang and
                  Wen{-}Chih Peng},
  title        = {A Survey on Self-Supervised Learning for Non-Sequential Tabular Data},
  journal      = {CoRR},
  volume       = {abs/2402.01204},
  year         = {2024}
}

Papers

Predictive Learning

• VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain (NeurIPS'20) [Paper] [Supplementary] [Code]
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (ACL'20) [Paper]
• TABBIE: Pretrained Representations of Tabular Data (NAACL'21) [Paper]) [Code]
• CORE: Self- and Semi-supervised Tabular Learning with COnditional REgularizations (NeurIPS'21) [Paper]
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings [Paper]
• TabNet: Attentive Interpretable Tabular Learning (AAAI'21) [Paper] Code
• Self-Supervision Enhanced Feature Selection with Correlated Gates (ICLR'22) [Paper] [Code]
• TransTab: Learning Transferable Tabular Transformers Across Tables (NeurIPS'22) [Paper] [Code] [Blog]
• LIFT: Language-Interfaced Fine-Tuning for Non-language Machine Learning Tasks (NeurIPS'22) [Paper] [Code]
• Self Supervised Pre-training for Large Scale Tabular Data (NeurIPS'22 TRL Workshop) [Paper] [Blog]
• Local Contrastive Feature Learning for Tabular Data (CIKM'22) [Paper]
• Revisiting Self-Training with Regularized Pseudo-Labeling for Tabular Data (preprint'23) [Paper]
• Generative Table Pre-training Empowers Models for Tabular Prediction (EMNLP'23) [Paper] [Code]
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (ICLR'23) [Paper] [Code]
• STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables (ICLR'23) [Paper] [Code]
• Language Models are Realistic Tabular Data Generators (ICLR'23) [Paper] [Code]
• Self-supervised Representation Learning from Random Data Projectors (NeurIPS'23 TRL Workshop) [Paper] [Code]
• SwitchTab: Switched Autoencoders Are Effective Tabular Learners (AAAI'24) [Paper]
• Making Pre-trained Language Models Great on Tabular Prediction (ICLR'24) [Paper]
• Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains (ICML'24) [Paper] [Code]
• Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS'24) [Paper] [Code]
• Accurate predictions on small data with a tabular foundation model (Nature'25) [Paper]
• T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data (ICLR'25) [Paper] [Code]
• MOTHERNET: Fast Training and Inference via Hyper-network Transformers (ICLR'25) [Paper] [Code]
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data (ICML'25) [Paper] [Code]
• TabDPT: Scaling Tabular Foundation Models on Real Data (NeurIPS'25) [Paper] [Code]
• MITRA: Mixed Synthetic Priors for Enhancing Tabular Foundation Models (NeurIPS'25) [Paper] [Model]

Contrastive Learning

• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption (ICLR'22) [Paper] [Code]
• STab: Self-supervised Learning for Tabular Data (NeurIPS'22 Workshop on TRL) [Paper]
• TransTab: Learning Transferable Tabular Transformers Across Tables (NeurIPS'22) [Paper]
• PTaRL: Prototype-based Tabular Representation Learning via Space Calibration (ICLR'24) [Paper]
• Tabular Data Contrastive Learning via Class-Conditioned and Feature-Correlation Based Augmentation (preprint) [Paper] [Code]
• TabDeco: A Comprehensive Contrastive Framework for Decoupled Representations in Tabular Data (NeurIPS'24 TRL Workshop) [Paper]
• ICE-T: Interactions-aware Cross-column Contrastive Embedding for Heterogeneous Tabular Datasets (NeurIPS'24 TRL Workshop) [Paper]

Hybrid Learning

• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning (NeurIPS'21) [Paper] [Supplementary] [Code]
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (NurIPS‘22 Workshop on TRL) [Paper] [Code]
• Transfer Learning with Deep Tabular Models (ICLR'23) [Paper] [Code]
• DoRA: Domain-Based Self-Supervised Learning Framework for Low-Resource Real Estate Appraisal (CIKM'23) [Paper] [Code]
• ReConTab: Regularized Contrastive Representation Learning for Tabular Data (NeurIPS'23 Workshop on TRL) [Paper]
• XTab: Cross-table Pretraining for Tabular Transformers (ICML'23) [Paper]
• UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR'24) [Paper]

Benchmarks

Tutorials

• Self-Supervised Learning: Self-Prediction and Contrastive Learning (NeurIPS'21) [Website]

Workshops

• Table Representation Learning Workshop (ACL'25) [Website]
• Foundation Models for Structured Data Workshop (ICML'25) [Website]

Related Survey

• Deep Neural Networks and Tabular Data: A Survey [Paper]
• Self-Supervised Learning for Recommender Systems: A Survey (TKDE) [Paper]
• Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data [Paper]
• Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects [Paper]
• On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence [Paper]
• A Survey on Time-Series Pre-Trained Models [Paper]
• Deep Learning within Tabular Data: Foundations, Challenges, Advances and Future Directions [Paper]
• Representation Learning for Tabular Data: A Comprehensive Survey [Paper]

Tools & Libraries

• Pytorch Frame: A modular deep learning framework for building neural network models on heterogeneous tabular data [Link]
• PyTorch Tabular: A Framework for Deep Learning with Tabular Data [Link]
• Pytorch wide-deep: A flexible package for multimodal-deep-learning to combine tabular data with text and images using Wide and Deep models in Pytorch [Link]