Unified Time Series Dataset (UTSD)

Updates

:triangular_flag_on_post: News (2024.10) We release the numpy format of UTSD. An easier and more efficient dataloader can be found here.

Introduction

We curate Unified Time Series Dataset (UTSD) that includes 7 domains with up to 1 billion time points with hierarchical four volumes to facilitate research of large models and pre-training in the field of time series.

Unified Time Series Dataset (UTSD) is meticulously assembled from a blend of publicly accessible online data repositories and empirical data derived from real-world machine operations.

All datasets are classified into seven distinct domains by their source: Energy, Environment, Health, Internet of Things (IoT), Nature, Transportation, and Web with diverse sampling frequencies.

See this paper for more information.

Usage

You can access and load UTSD based on this GitHub.

# huggingface-cli login
# export HF_ENDPOINT=https://hf-mirror.com 

python ./scripts/UTSD/download_dataset.py

# dataloader
python ./scripts/UTSD/utsdataset.py

It should be noted that due to the construction of our dataset with diverse lengths, the sequence lengths of different samples vary. You can construct the data organization logic according to your own needs.

In addition, we provide code dataset_evaluation.py for evaluating time series datasets, which you can use to evaluate your Huggingface formatted dataset. The usage of this script is as follows:

python ./scripts/UTSD/dataset_evaluation.py --root_path <dataset root path> --log_path <output log path>

Dataset Description

We analyze each dataset, examining the time series through the lenses of stationarity and forecastability to allow us to characterize the level of complexity inherent to each dataset.

You can find the specific source address in source.csv.

[1]: Monash Time Series Forecasting Archive [2]: Time series extrinsic regression Predicting numeric values from time series data [3]: The UCR Time Series Archive [4]: ERA5-Land: a state-of-the-art global reanalysis dataset for land applications [5]: Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series

Hierarchy of Datasets

UTSD is constructed with hierarchical capacities, namely UTSD-1G, UTSD-2G, UTSD-4G, and UTSD-12G, where each smaller dataset is a subset of the larger ones. A larger subset means greater data difficulty and diversity, allowing you to conduct detailed scaling experiments.

Acknowledgments

UTSD is mostly built from the Internet public time series dataset, which comes from different research teams and providers. We sincerely thank all individuals and organizations who have contributed the data. Without their generous sharing, this dataset would not have existed.

Citation

If you're using UTSD in your research or applications, please cite it using this BibTeX:

BibTeX:

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}