Is this chart lying to me? Automating the detection of misleading visualizations

This repository contains the datasets and code associated with the ACL 2026 Main conference paper: Is this chart lying to me? Automating the detection of misleading visualizations. The Misviz and Misviz-synth datasets are released under a CC-BY-SA 4.0 license. The code is released under an Apache 2.0 license.

Contact person: Jonathan Tonglet

UKP Lab | TU Darmstadt

Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.

News

• April 2026: Our paper has been accepted to ACL 2026 Main conference 🔥 See you in San Diego 🌊
• March 2026: Misviz is now available on HuggingFace
• February 2026: Check out our new paper on misleading visualizations: ChartAttack
• January 2026: We released a new version of Misviz that contains bounding box coordinates for a subset of the misleaders
• January 2026: We updated the preprint and released a new version of Misviz-synth that contains maps and scatterplots

Abstract

Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccurate conclusions. Prior work has shown that both humans and multimodal large language models (MLLMs) are frequently deceived by such visualizations. Automatically detecting misleading visualizations and identifying the specific design rules they violate could help protect readers and reduce the spread of misinformation. However, the training and evaluation of AI models has been limited by the absence of large, diverse, and openly available datasets. In this work, we introduce Misviz, a benchmark of 2,604 real-world visualizations annotated with 12 types of misleaders. To support model training, we also release Misviz-synth, a synthetic dataset of 51,665 visualizations generated using Matplotlib and based on real-world data tables. We perform a comprehensive evaluation on both datasets using state-of-the-art MLLMs, rule-based systems, and fine-tuned classifiers. Our results reveal that the task remains highly challenging.

tl;dr

😯 Misleading visualizations are a dangerous form of misinformation. We need to develop methods to detect design issues in charts that make them misleading

📊 We introduce two datasets for the task of misleading visualization detection

• Misviz-synth: a large dataset of synthetic matplotlib charts based on real-world data tables
• Misviz: a small benchmark dataset of real-world charts collected from the web
• Instructions to load the datasets are provided in Datasets

🤖 We evaluate three type of models on both datasets

• Zero-shot MLLMs
• A new rule-based linter that verifies the axis metadata of a visualization
• Two classifiers fine-tuned on Misviz-synth

💡Our experiment results reveal several key insights

• The task is challenging both on synthetic and real-world visualizations even for SOTA MLLMs
• Fine-tuned classifiers can generalize to some extent to real-world visualizations
• The axis extraction phase is very error prone on real-world visualizations

Datasets

We briefly describe the datasets below. More information can be found in the README of the data folder.

Misviz-synth

• data/misviz_synth/misviz_synth.json contains the task labels and metadata
• The visualizations, the underlying data tables, the code snippets, and the axis metadata can be downloaded from TUdatalib

Misviz

• data/misviz/misviz.json contains the task labels and metadata
• The visualizations can be downloaded from the web using the following script. Please contact the authors if you face any issues downloading the images.

python data/download_misviz_images.py --use_wayback 0

• The dataset can also be accessed on HuggingFace. However, please keep in mind that the experiment code is designed for the JSON version of the dataset available in this repo.

Misviz instance example

  {
      "image_path": "img/68718369730_misrepresentation.png",
      "image_url": "https://64.media.tumblr.com/88844d8c3be687e0549e7b7c0a403293/tumblr_mx1as48rLr1sgh0voo1_1280.jpg",
      "chart_type": [
          "bar chart",
          "pie chart"
      ],
      "misleader": [
          "misrepresentation"
      ],
      "wayback_image_url": "https://web.archive.org/web/20250619095605/https://64.media.tumblr.com/88844d8c3be687e0549e7b7c0a403293/tumblr_mx1as48rLr1sgh0voo1_1280.jpg",
      "split": "test",
      "bbox": []
  }

Environment

Follow these instructions to recreate the environment used for our experiments.

$ conda create --name lying_charts python=3.10
$ conda activate lying_charts
$ pip install -r requirements.txt

Experiments

Evaluate zero-shot MLLMs

python src/mllm_inference/misleader_detection_MLLM.py --datasets misviz_synth-misviz --split test --model internvl3/8B/ --max_tokens 200

The --model argument expects a string in the format model_name/model_size/. By default, the following models are available:

We also provide code to run experiments with GPT-4.1, GPT-o3, and Gemini-2.5-flash-lite using the OpenAI API and Google AI Studio. You will first need to obtain API keys from both providers and store them as environment variables.

Fine-tune DePlot for axis extraction and predict axis metadata

To apply the linter to Misviz and to train the classifiers, we need to extract axis metadata from visualizations. This is done by fine-tuning DePlot on the Misviz-synth train set. To fine-tune deplot, we provide a shell script which you can adjust to your needs.

sbatch src/model_tuning/02_deplot_finetune/01_run_accelerate_deplot_finetuning.sh

Once the model is fine-tuned, axes can be predicted for all splits of Misviz-synth and Misviz using the following two shell scripts, which you can again adjust to your needs.

sbatch src/model_tuning/03_deplot_axis_extraction_classifier/01_run_axis_prediction_precomp_split_0.sh
sbatch src/model_tuning/03_deplot_axis_extraction_classifier/01_run_axis_prediction_precomp_split_1.sh

Rule-based linter

The rule-based linter can be evaluated both on ground truth and predicted axis metadata for Misviz-synth, but only on predicted axis metadata for Misviz.

python src/rule_based_linter/linter.py --datasets misviz_synth --split test --use_predicted_axis 0

Fine-tuned classifiers

To train the classifiers, the embeddings for the visualization images and the axis metadata can be precomputed using TinyChart and TAPAS, respectively.

For image embeddings, adjust and run the following shell script:

$ sbatch src/model_tuning/01_precomputation/01_run_all_img_precomp.sh

For axis metadata embeddings, run the following Python script.

$ sbatch src/model_tuning/03_deplot_axis_extraction_classifier/02_encode_tables.sh

Then, the classifiers can be trained as follows:

$ sbatch src/model_tuning/03_deplot_axis_extraction_classifier/03_run_all_experiments.sh

To make inferences with the trained classifiers, use the following script:

$ sbatch src/model_tuning/03_deplot_axis_extraction_classifier/04_inference.sh

Citation

If you find this work relevant to your research or use this code in your work, please cite our paper as follows:

@inproceedings{tonglet2026misviz,
  title={Is this chart lying to me? Automating the detection of misleading visualizations},
  author={Tonglet, Jonathan and Zimny, Jan and Tuytelaars, Tinne and Gurevych, Iryna},
  booktitle={The 64th Annual Meeting of the Association for Computational Linguistics},
  year={2026},
  url={https://arxiv.org/abs/2508.21675},
  doi={10.48550/arXiv.2508.21675}
}

Disclaimer

This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.