Awesome-Time-Series-Explainability

A list of XAI for time series. This list focuses (currently) on Post-Hoc Explainability for time series data, including paper and github links. The list is expanded and updated gradually. Feel Free to update missing or new paper.

Outline

• Surveys
• Libraries
• Classification
• Regression / Forecasting
• Classification and Regression / Forcasting
• Benchmarking and Evaluation
• Ante-Hoc Explanations

Surveys

• Explainable artificial intelligence (XAI) in finance: a systematic literature review , (2024) by Černevičienė, J., & Kabašinskas, A
• A Survey of Explainable Artificial Intelligence (XAI) in Financial Time Series Forecasting , (2024) by Arsenault, P. D., Wang, S., & Patenande, J. M.
• Post Hoc Explainability for Time Series Classification: Toward a signal processing perspective , (2022) by R. Mochaourab, A. Venkitaraman, I. Samsten, P. Papapetrou and C. R. Rojas
• Explainable AI for time series classification: a review, taxonomy and research directions , (2022) by A Theissler, F Spinnato, U Schlegel, R Guidotti
• Explainable artificial intelligence (xai) on timeseries data: A survey , (2021) by Rojat, T., Puget, R., Filliat, D., Del Ser, J., Gelin, R., & Díaz-Rodríguez, N.
• XAI Methods for Neural Time Series Classification: A Brief Review , (2021) by Simic, I., Sabol, V., Veas, E.

Libraries

• TSInterpret: A Python Package for the Interpretability of Time Series Classification (2023) by Höllig, J., Kulbach, C., & Thoma, S. https://github.com/fzi-forschungszentrum-informatik/TSInterpret,
• Time Interpret: a Unified Model Interpretability Library for Time Series (2023) by Enguehard, J. https://github.com/josephenguehard/time_interpret,
• TSGap: Composable time-series missingness simulation. Separates mechanisms (MCAR/MAR/MNAR) from patterns (pointwise/block/monotone/decay/markov) (2025) by Oripov, F. https://github.com/feruzoripov/tsgap,

Classification

Feature Attribution

• Time is Not Enough: Time-Frequency based Explanation for Time-Series Black-Box Models, (2024), by *Chung, H., Jo, S., Kwon, Y., & Choi, E. * https://github.com//gustmd0121/time_is_not_enough,
• Translating Image XAI to Multivariate Time Series, (2024), by Tronchin, L., Cordelli, E., Celsi, L. R., Maccagnola, D., Natale, M., Soda, P., & Sicilia, R. https://github.com//ltronchin/translating-xai-mts,
• Explaining time series classifiers through meaningful perturbation and optimisation, (2023), by H Meng, C Wagner, I Triguero https://github.com/menghan1994/ETSC_through_Meainingful_Perturbation_and_Optimisation,
• Explainable AI for Time Series via Virtual Inspection Layers , (2023) by Vielhaben, J., Lapuschkin, S., Montavon, G., & Samek, W.
• LIMESegment: Meaningful, Realistic Time Series Explanations , (2022) by Sivill, T., & Flach, P., https://github.com/TortySivill/LIMESegment,
• Class-Specific Explainability for Deep Time Series Classifiers , (2022) by Doddaiah, R., Parvatharaju, P., Rundensteiner, E., & Hartvigsen, T., https://github.com/rameshdoddaiah/DEMUX,
• TSInsight: A local-global attribution framework for interpretability in time series data(2021) by Siddiqui, S. A., Mercier, D., Dengel, A., & Ahmed, S.
• Benchmarking Deep Learning Interpretability in Time Series Predictions (2020) by Ismail, A. A., Gunady, M., Corrada Bravo, H., & Feizi, S. https://github.com/ayaabdelsalam91/TS-Interpretability-Benchmark,
• What went wrong and when? Instance-wise feature importance for time-series black-box models (2020) by Tonekaboni, S., Joshi, S., Campbell, K., Duvenaud, D. K., & Goldenberg, A. https://github.com/sanatonek/time_series_explainability,
• Agnostic Local Explanation for Time Series Classification (2019) by *Guillemé, M., Masson, V., Rozé, L., & Termier, A. *
• timeXplain -- A Framework for Explaining the Predictions of Time Series Classifiers (2019) by Mujkanovic, F., Doskoč, V., Schirneck, M., Schäfer, P., & Friedrich, T. https://github.com/LoadingByte/timeXplain,
• Tsxplain: Demystification of dnn decisions for time-series using natural language and statistical features (2019) by Munir, M., Siddiqui, S. A., Küsters, F., Mercier, D., Dengel, A., & Ahmed, S.

Counterfactuals

• Sub-SpaCE: Subsequence-Based Sparse Counterfactual Explanations for Time Series Classification Problems (2024) by Refoyo, M., & Luengo, D. https://github.com/MarioRefoyo/Sub-SpaCE,
• CELS: Counterfactual Explanations for Time Series Data via Learned Saliency Maps (2023) by Li, P., Bahri, O., Boubrahimi, S. F., & Hamdi, S. M. https://github.com/Luckilyeee/CELS,
• Glacier: guided locally constrained counterfactual explanations for time series classification (2024) by Wang, Z., Samsten, I., Miliou, I., Mochaourab, R., & Papapetrou, P. https://github.com/zhendong3wang/learning-time-series-counterfactuals,
• Attention-Based Counterfactual Explanation for Multivariate Time Series (2023) by Li, P., Boubrahimi, S. F., & Hamdi, S. M. https://sites.google.com/view/attention-based-cf
• Shapelet-Based Counterfactual Explanations for Multivariate Time Series (2022) by Bahri, O., Boubrahimi, S. F., & Hamdi, S. M. https://github.com/omarbahri/SETS,
• TSEvo: Evolutionary counterfactual explanations for time series classification (2022) by Höllig, J., Kulbach, C., & Thoma, S. https://github.com/JHoelli/TSEvo,
• Counterfactual explanations for multivariate time series (2021) by Ates, E., Aksar, B., Leung, V. J., & Coskun, A. K. https://github.com/peaclab/CoMTE,
• Motif-Guided Time Series Counterfactual Explanations (2022) by Li, P., Boubrahimi, S. F., & Hamdi, S. M. https://github.com/Luckilyeee/Motif-guided-counterfactual-explanation,
• Instance-based Counterfactual Explanations for Time Series Classification (2020) by Delaney, E., Greene, D., & Keane, M. T. https://github.com/e-delaney/Instance-Based_CFE_TSC,

Frameworks (Multiple Explanation Types)

• ExTea: An Evolutionary Algorithm-Based Approach for Enhancing Explainability in Time-Series Models (2024) by Huang, Y., Zhou, Y., Zhao, H., Fang, L., Riedel, T., & Beigl, M https://github.com/HuangYiran/extea,
• Understanding Any Time Series Classifier with a Subsequence-based Explainer , (2023) by Spinnato, F., Guidotti, R., Monreale, A., Nanni, M., Pedreschi, D., & Giannotti, F. https://github.com/fspinna/lasts,
• TimeSHAP: Explaining Recurrent Models through Sequence Perturbations, (2020) by Bento, J., Saleiro, P., Cruz, A. F., Figueiredo, M. A., & Bizarro, P., https://github.com/feedzai/timeshap,

Forecasting

• ShapTime: A General XAI Approach for Explainable Time Series Forecasting (2024) by Zhang, Y., Sun, Q., Qi, D., Liu, J., Ma, R., & Petrosian, O. https://github.com/Zhangyuyi-0825/ShapTime,
• TsSHAP: Robust model agnostic feature-based explainability for univariate time series forecasting (2023) by Raykar, V. C., Jati, A., Mukherjee, S., Aggarwal, N., Sarpatwar, K., Ganapavarapu, G., & Vaculin, R.
• Counterfactual Explanations for Time Series Forecasting (2023) by Wang, Z., Miliou, I., Samsten, I., & Papapetrou, P. https://github.com/zhendong3wang/counterfactual-explanations-for-forecasting,
• TEMPORAL DEPENDENCIES IN FEATURE IMPORTANCE FOR TIME SERIES PREDICTION (2023) by Leung, K. K., Rooke, C., Smith, J., Zuberi, S., & Volkovs, M. https://github.com/layer6ai-labs/WinIT,
• TS-MULE: Local Interpretable Model-Agnostic Explanations for Time Series Forecast Models (2021) by Schlegel, U., Vo, D. L., Keim, D. A., & Seebacher, D. https://github.comdbvis-ukon/ts-mule,
• Explaining time series predictions with dynamic masks (2021) by Crabbé, J., & Van Der Schaar, M. https://github.com/JonathanCrabbe/Dynamask,
• Series saliency: Temporal interpretation for multivariate time series forecasting (2020) by Pan, Q., Hu, W., & Zhu, J.

Classification and Regression / Forcasting

• Tsviz: Demystification of deep learning models for time-series analysis (2019) by Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. https://github.com/shoaibahmed/TSViz-Core,
• Learning Perturbations to Explain Time Series Predictions (2023) by Enguehard, J. https://github.com/josephenguehard/time_interpret,

Benchmarking and Evaluation

• XAI for Time Series Classification: Evaluating the Benefits of Model Inspection for End-Users, (2024) by Håvardstun, B., Ferri, C., Flikka, K., & Telle, J. A
• A Deep Dive into Perturbations as Evaluation Technique for Time Series XAI, (2023) by Schlegel, U., & Keim, D. A., https://github.com/visual-xai-for-time-series/time-series-xai-perturbation-analysis,
• Evaluation of post-hoc interpretability methods in time-series classification, (2023) by Turbé, H., Bjelogrlic, M., Lovis, C. et al., https://github.com/hturbe/InterpretTime,
• A Deep Dive into Perturbations as Evaluation Technique for Time Series XAI, (2023) by Schlegel, U., & Keim, D. A.
• Introducing the Attribution Stability Indicator: a Measure for Time Series XAI Attributions, (2023) by *Schlegel, U., & Keim, D. A. *
• Robust Framework for Explanation Evaluation in Time Series Classification, (2023) by Nguyen, T. T., Nguyen, T. L., & Ifrim, G. https://github.com/mlgig/amee,
• Evaluating Explanation Methods for Multivariate Time Series Classification, (2023) by *Serramazza, D. I., Nguyen, T. T., Nguyen, T. L., & Ifrim, G. * https://github.com/mlgig/Evaluating-Explanation-Methods-for-MTSC,
• XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification, (2023) by Höllig, J., Thoma, S., & Grimm, F. https://github.com/JHoelli/XTSC-Bench,
• Time to Focus: A Comprehensive Benchmark Using Time Series Attribution Methods, (2022) by Mercier, D., Bhatt, J., Dengel, A., & Ahmed, S.
• Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series, (2021) by Jacob, V., Song, F., Stiegler, A., Rad, B., Diao, Y., & Tatbul, N. https://github.com/exathlonbenchmark/exathlon,
• Towards a rigorous evaluation of XAI methods on time series, (2019) by Schlegel, U., Arnout, H., El-Assady, M., Oelke, D., & Keim, D. A.

Ante-Hoc Explanation

While this repository mostly focuses on post-hoc explanations - i.e. introducing the explanations after the training of the predictor, this section includes some approches that include explanability into the predicors design (e.g., via architecture or training).

Classification

• Fast, accurate and explainable time series classification through randomization (2023) by Cabello, N., Naghizade, E., Qi, J., & Kulik, L. https://github.com/stevcabello/r-STSF,
• XEM: An explainable-by-design ensemble method for multivariate time series classification (2022) by Fauvel, K., Fromont, É., Masson, V., Faverdin, P., & Termier, A https://github.com/XAIseries/XEM,
• Xcm: An explainable convolutional neural network for multivariate time series classification (2021) by Fauvel, K., Lin, T., Masson, V., Fromont, É., & Termier, A. https://github.com/XAIseries/XCM,
• Explaining Deep Classification of Time-Series Data with Learned Prototypes (2019) by Gee, A. H., Garcia-Olano, D., Ghosh, J., & Paydarfar, D. https://github.com/alangee/ijcai19-ts-prototypes,
• Medical Time Series Classification with Hierarchical Attention-based Temporal Convolutional Networks: A Case Study of Myotonic Dystrophy Diagnosis (2019) by Lin, L., Xu, B., Wu, W., Richardson, T. W., & Bernal, E. A.
• Explainable Failure Predictions with RNN Classifiers based on Time Series Data (2019) by Giurgiu, I., & Schumann, A.
• MTEX-CNN: Multivariate Time Series EXplanations for Predictions with Convolutional Neural Networks (2019) by Assaf, R., Giurgiu, I., Bagehorn, F., & Schumann, A.
• Retain: An interpretable predictive model for healthcare using reverse time attention mechanism (2016) by Choi, E., Bahadori, M. T., Sun, J., Kulas, J., Schuetz, A., & Stewart, W. https://github.com/mp2893/retain,

Forecasting

• Temporal fusion transformers for interpretable multi-horizon time series forecasting (2021) by Lim, B., Arık, S. Ö., Loeff, N., & Pfister, T.
• Interpretable Multivariate Time Series Forecasting with Temporal Attention Convolutional Neural Networks (2020) by Pantiskas, L., Verstoep, K., & Bal, H. https://github.com/lpphd/multivariate-attention-tcn,
• Exploring interpretable LSTM neural networks over multi-variable data (2019) by Guo, Tian, Tao Lin, and Nino Antulov-Fantulin https://github.com/weilai0980/IMV-LSTM,
• A memory-network based solution for multivariate time-series forecasting (2018) by Chang, Y. Y., Sun, F. Y., Wu, Y. H., & Lin, S. D.