README
Awesome Text VQA
Text related VQA is a fine-grained direction of the VQA task, which only focuses on the question that requires to read the textual content shown in the input image.
Datasets
- NewsVideoQA dataset (WACV2023) [Project][Paper]
- ViteVQA dataset (NeurIPS 2022) [Project][Paper]
- VisualMRC dataset (AAAI 2021) [Project][Paper]
- EST-VQA dataset (CVPR 2020) [Project][Paper]
- DOC-VQA dataset (CVPR Workshop 2020) [Project][Paper]
- Text-VQA dataset (CVPR 2019) [Project][Paper]
- ST-VQA dataset (ICCV 2019) [Project][Paper]
- OCR-VQA dataset (ICDAR 2019) [Project][Paper]
| Dataset | #Train+Val Img | #Train+Val Que | #Test Img | #Test Que | Image Source | Language |
|---|---|---|---|---|---|---|
| Text-VQA | 25,119 | 39,602 | 3,353 | 5,734 | [1] | EN |
| ST-VQA | 19,027 | 26,308 | 2,993 | 4,163 | [2, 3, 4, 5, 6, 7, 8] | EN |
| OCR-VQA | 186,775 | 901,717 | 20,797 | 100,429 | [9] | EN |
| EST-VQA | 17,047 | 19,362 | 4,000 | 4,525 | [4, 5, 8, 10, 11, 12, 13] | EN+CH |
| DOC-VQA | 11,480 | 44,812 | 1,287 | 5,188 | [14] | EN |
| VisualMRC | 7,960 | 23,854 | 2,237 | 6,708 | self-collected webpage screenshot | EN |
| ViteVQA(Task1Spilt1) | 5,969 | 19,840 | 971 | 3,183 | YouTuBe | EN |
Image Source:
[1] OpenImages: A public dataset for large-scale multi-label and multi-class image classification (v3) [dataset]
[2] Imagenet: A large-scale hierarchical image database [dataset]
[3] Vizwiz grand challenge: Answering visual questions from blind people [dataset]
[4] ICDAR 2013 robust reading competition [dataset]
[5] ICDAR 2015 competition on robust reading [dataset]
[6] Visual Genome: Connecting language and vision using crowdsourced dense image annotations [dataset]
[7] Image retrieval using textual cues [dataset]
[8] Coco-text: Dataset and benchmark for text detection and recognition in natural images [dataset]
[9] Judging a book by its cover [dataset]
[10] Total Text [dataset]
[11] SCUT-CTW1500 [dataset]
[12] MLT [dataset]
[13] Chinese Street View Text [dataset]
[14] UCSF Industry Document Library [dataset]
Related Challenges
ICDAR 2021 COMPETITION On Document Visual Question Answering (DocVQA) Submission Deadline: 31st March 2021 [Challenge]
Document Visual Question Answering （CVPR 2020 Workshop on Text and Documents in the Deep Learning Era Submission Deadline: 30 April 2020 [Challenge]
Papers
2023
- [RUArt] RUArt: A Novel Text-Centered Solution for Text-Based Visual Question Answering(T-MM) [Paper][Project]
- [BOV++] Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa (PR) [Paper]
2022
- [TAG] TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation (PR) [Paper][Project]
- [ViteVQA] Towards Video Text Visual Question Answering: Benchmark and Baseline (NeurIPS) [Paper][Project]
- [LaTr] LaTr: Layout-Aware Transformer for Scene-Text VQA (CVPR) [Paper][Unofficial Code]
- [TIG] Text-instance graph: Exploring the relational semantics for text-based visual question answering (PR)[Paper]
- [SMA] Structured Multimodal Attentions for TextVQA (T-PAMI)[Paper][Project]
- [DA-Net] Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering (arXiv)[Paper]
- [SenseGATE] SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering (arXiv)[Paper]
- [MLCI] Multi-level, multi-modal interactions for visual question answering over text in images (WWW)[Paper][Project]
- [TWA] From Token to Word: OCR Token Evolution via Contrastive Learning and Semantic Matching for Text-VQA (ACMMM)[Paper][Project]
- [TAG] TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation (BMVC)[Paper][Project]
- [MGEN] MODALITY-SPECIFIC MULTIMODAL GLOBAL ENHANCED NETWORK FOR TEXT-BASED VISUAL QUESTION ANSWERING (ICME)[Paper]
- [SC-Net] TOWARDS ESCAPING FROM LANGUAGE BIAS AND OCR ERROR: SEMANTICS-CENTERED TEXT VISUAL QUESTION ANSWERING (ICME)[Paper]
- [EKTVQA] EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA (Access)[Paper]
- [Two-stage fusion] Two-stage Multimodality Fusion for High-performance Text-based Visual Question Answering (ACCV)[Paper]
2021
- [VisualMRC] VisualMRC: Machine Reading Comprehension on Document Images (AAAI) [Paper][Project]
- [SSBaseline] Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps (AAAI) [Paper][code]
2020
- [SA-M4C] Spatially Aware MultimodalTransformers for TextVQA (ECCV) [Paper][Project][Code]
- [EST-VQA] On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering (CVPR) [Paper]
- [M4C] Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA (CVPR) [Paper][Project]
- [LaAP-Net] Finding the Evidence: Localization-aware Answer Prediction for TextVisual Question Answering (COLING) [Paper]
- [CRN] Cascade Reasoning Network for Text-basedVisual Question Answering (ACM MM) [Paper][Project]
2019
- [Text-VQA/LoRRA] Towards VQA Models That Can Read (CVPR) [Paper][Code]
- [ST-VQA] Scene Text Visual Question Answering (ICCV) [Paper]
- [Text-KVQA] From Strings to Things: Knowledge-enabled VQA Modelthat can Read and Reason (ICCV) [Paper]
- [OCR-VQA] OCR-VQA: Visual Question Answering by Reading Text in Images (ICDAR) [Paper]
Technical Reports
- [DiagNet] DiagNet: Bridging Text and Image [Report][Code]
- [DCD_ZJU] Winner of 2019 Text-VQA challenge [Slides]
- [Schwail] Runner-up of 2019 Text-VQA challenge [Slides]
Benchmark
Acc. : Accuracy I. E. : Image Encoder Q. E. : Question Encoder O. E. : OCR Token Encoder Ensem. : Ensemble
Text-VQA
[official leaderboard(2019)] [official leaderboard(2020)]
| Y-C./J. | Methods | Acc. | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2019--CVPR | LoRRA | 26.64 | Faster R-CNN | GloVe | Rosetta-ml | FastText | Classification | N |
| 2019--N/A | DCD_ZJU | 31.44 | Faster R-CNN | BERT | Rosetta-ml | FastText | Classification | Y |
| 2020--CVPR | M4C | 40.46 | Faster R-CNN (ResNet-101) | BERT | Rosetta-en | FastText | Decoder | N |
| 2020--Challenge | Xiangpeng | 40.77 | ||||||
| 2020--Challenge | colab_buaa | 44.73 | ||||||
| 2020--Challenge | CVMLP(SAM) | 44.80 | ||||||
| 2020--Challenge | NWPU_Adelaide_Team(SMA) | 45.51 | Faster R-CNN | BERT | BDN | Graph Attention | Decoder | N |
| 2020--ECCV | SA-M4C | 44.6* | Faster R-CNN (ResNext-152) | BERT | Google-OCR | FastText+PHOC | Decoder | N |
| 2020--arXiv | TAP | 53.97* | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
| 2022--arXiv | TAG | 53.63 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
* Using external data for training.
ST-VQA
[official leaderboard]
T1 : Strongly Contextualised Task
T2 : Weakly Contextualised Task
T3 : Open Dictionary
| Y-C./J. | Methods | Acc. (T1/T2/T3) | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2020--CVPR | M4C | na/na/0.4621 | Faster R-CNN (ResNet-101) | BERT | Rosetta-en | FastText | Decoder | N |
| 2020--Challenge | SMA | 0.5081/0.3104/0.4659 | Faster | BERT | BDN | Graph Attention | Decoder | N |
| 2020--ECCV | SA-M4C | na/na/0.5042 | Faster R-CNN (ResNext-152) | BERT | Google-OCR | FastText+PHOC | Decoder | N |
| 2020--arXiv | TAP | na/na/0.5967 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
| 2022--arXiv | TAG | na/na/0.6019 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
OCR-VQA
| Y-C./J. | Methods | Acc. | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2020--CVPR | M4C | 63.9 | Faster R-CNN | BERT | Rosetta-en | FastText | Decoder | N |
Contributors
27 commits
1 commits
Similar repos
README
Awesome Text VQA
Text related VQA is a fine-grained direction of the VQA task, which only focuses on the question that requires to read the textual content shown in the input image.
Datasets
- NewsVideoQA dataset (WACV2023) [Project][Paper]
- ViteVQA dataset (NeurIPS 2022) [Project][Paper]
- VisualMRC dataset (AAAI 2021) [Project][Paper]
- EST-VQA dataset (CVPR 2020) [Project][Paper]
- DOC-VQA dataset (CVPR Workshop 2020) [Project][Paper]
- Text-VQA dataset (CVPR 2019) [Project][Paper]
- ST-VQA dataset (ICCV 2019) [Project][Paper]
- OCR-VQA dataset (ICDAR 2019) [Project][Paper]
| Dataset | #Train+Val Img | #Train+Val Que | #Test Img | #Test Que | Image Source | Language |
|---|---|---|---|---|---|---|
| Text-VQA | 25,119 | 39,602 | 3,353 | 5,734 | [1] | EN |
| ST-VQA | 19,027 | 26,308 | 2,993 | 4,163 | [2, 3, 4, 5, 6, 7, 8] | EN |
| OCR-VQA | 186,775 | 901,717 | 20,797 | 100,429 | [9] | EN |
| EST-VQA | 17,047 | 19,362 | 4,000 | 4,525 | [4, 5, 8, 10, 11, 12, 13] | EN+CH |
| DOC-VQA | 11,480 | 44,812 | 1,287 | 5,188 | [14] | EN |
| VisualMRC | 7,960 | 23,854 | 2,237 | 6,708 | self-collected webpage screenshot | EN |
| ViteVQA(Task1Spilt1) | 5,969 | 19,840 | 971 | 3,183 | YouTuBe | EN |
Image Source:
[1] OpenImages: A public dataset for large-scale multi-label and multi-class image classification (v3) [dataset]
[2] Imagenet: A large-scale hierarchical image database [dataset]
[3] Vizwiz grand challenge: Answering visual questions from blind people [dataset]
[4] ICDAR 2013 robust reading competition [dataset]
[5] ICDAR 2015 competition on robust reading [dataset]
[6] Visual Genome: Connecting language and vision using crowdsourced dense image annotations [dataset]
[7] Image retrieval using textual cues [dataset]
[8] Coco-text: Dataset and benchmark for text detection and recognition in natural images [dataset]
[9] Judging a book by its cover [dataset]
[10] Total Text [dataset]
[11] SCUT-CTW1500 [dataset]
[12] MLT [dataset]
[13] Chinese Street View Text [dataset]
[14] UCSF Industry Document Library [dataset]
Related Challenges
ICDAR 2021 COMPETITION On Document Visual Question Answering (DocVQA) Submission Deadline: 31st March 2021 [Challenge]
Document Visual Question Answering （CVPR 2020 Workshop on Text and Documents in the Deep Learning Era Submission Deadline: 30 April 2020 [Challenge]
Papers
2023
- [RUArt] RUArt: A Novel Text-Centered Solution for Text-Based Visual Question Answering(T-MM) [Paper][Project]
- [BOV++] Beyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa (PR) [Paper]
2022
- [TAG] TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation (PR) [Paper][Project]
- [ViteVQA] Towards Video Text Visual Question Answering: Benchmark and Baseline (NeurIPS) [Paper][Project]
- [LaTr] LaTr: Layout-Aware Transformer for Scene-Text VQA (CVPR) [Paper][Unofficial Code]
- [TIG] Text-instance graph: Exploring the relational semantics for text-based visual question answering (PR)[Paper]
- [SMA] Structured Multimodal Attentions for TextVQA (T-PAMI)[Paper][Project]
- [DA-Net] Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering (arXiv)[Paper]
- [SenseGATE] SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering (arXiv)[Paper]
- [MLCI] Multi-level, multi-modal interactions for visual question answering over text in images (WWW)[Paper][Project]
- [TWA] From Token to Word: OCR Token Evolution via Contrastive Learning and Semantic Matching for Text-VQA (ACMMM)[Paper][Project]
- [TAG] TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation (BMVC)[Paper][Project]
- [MGEN] MODALITY-SPECIFIC MULTIMODAL GLOBAL ENHANCED NETWORK FOR TEXT-BASED VISUAL QUESTION ANSWERING (ICME)[Paper]
- [SC-Net] TOWARDS ESCAPING FROM LANGUAGE BIAS AND OCR ERROR: SEMANTICS-CENTERED TEXT VISUAL QUESTION ANSWERING (ICME)[Paper]
- [EKTVQA] EKTVQA: Generalized Use of External Knowledge to Empower Scene Text in Text-VQA (Access)[Paper]
- [Two-stage fusion] Two-stage Multimodality Fusion for High-performance Text-based Visual Question Answering (ACCV)[Paper]
2021
- [VisualMRC] VisualMRC: Machine Reading Comprehension on Document Images (AAAI) [Paper][Project]
- [SSBaseline] Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps (AAAI) [Paper][code]
2020
- [SA-M4C] Spatially Aware MultimodalTransformers for TextVQA (ECCV) [Paper][Project][Code]
- [EST-VQA] On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering (CVPR) [Paper]
- [M4C] Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA (CVPR) [Paper][Project]
- [LaAP-Net] Finding the Evidence: Localization-aware Answer Prediction for TextVisual Question Answering (COLING) [Paper]
- [CRN] Cascade Reasoning Network for Text-basedVisual Question Answering (ACM MM) [Paper][Project]
2019
- [Text-VQA/LoRRA] Towards VQA Models That Can Read (CVPR) [Paper][Code]
- [ST-VQA] Scene Text Visual Question Answering (ICCV) [Paper]
- [Text-KVQA] From Strings to Things: Knowledge-enabled VQA Modelthat can Read and Reason (ICCV) [Paper]
- [OCR-VQA] OCR-VQA: Visual Question Answering by Reading Text in Images (ICDAR) [Paper]
Technical Reports
- [DiagNet] DiagNet: Bridging Text and Image [Report][Code]
- [DCD_ZJU] Winner of 2019 Text-VQA challenge [Slides]
- [Schwail] Runner-up of 2019 Text-VQA challenge [Slides]
Benchmark
Acc. : Accuracy I. E. : Image Encoder Q. E. : Question Encoder O. E. : OCR Token Encoder Ensem. : Ensemble
Text-VQA
[official leaderboard(2019)] [official leaderboard(2020)]
| Y-C./J. | Methods | Acc. | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2019--CVPR | LoRRA | 26.64 | Faster R-CNN | GloVe | Rosetta-ml | FastText | Classification | N |
| 2019--N/A | DCD_ZJU | 31.44 | Faster R-CNN | BERT | Rosetta-ml | FastText | Classification | Y |
| 2020--CVPR | M4C | 40.46 | Faster R-CNN (ResNet-101) | BERT | Rosetta-en | FastText | Decoder | N |
| 2020--Challenge | Xiangpeng | 40.77 | ||||||
| 2020--Challenge | colab_buaa | 44.73 | ||||||
| 2020--Challenge | CVMLP(SAM) | 44.80 | ||||||
| 2020--Challenge | NWPU_Adelaide_Team(SMA) | 45.51 | Faster R-CNN | BERT | BDN | Graph Attention | Decoder | N |
| 2020--ECCV | SA-M4C | 44.6* | Faster R-CNN (ResNext-152) | BERT | Google-OCR | FastText+PHOC | Decoder | N |
| 2020--arXiv | TAP | 53.97* | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
| 2022--arXiv | TAG | 53.63 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
* Using external data for training.
ST-VQA
[official leaderboard]
T1 : Strongly Contextualised Task
T2 : Weakly Contextualised Task
T3 : Open Dictionary
| Y-C./J. | Methods | Acc. (T1/T2/T3) | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2020--CVPR | M4C | na/na/0.4621 | Faster R-CNN (ResNet-101) | BERT | Rosetta-en | FastText | Decoder | N |
| 2020--Challenge | SMA | 0.5081/0.3104/0.4659 | Faster | BERT | BDN | Graph Attention | Decoder | N |
| 2020--ECCV | SA-M4C | na/na/0.5042 | Faster R-CNN (ResNext-152) | BERT | Google-OCR | FastText+PHOC | Decoder | N |
| 2020--arXiv | TAP | na/na/0.5967 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
| 2022--arXiv | TAG | na/na/0.6019 | Faster R-CNN (ResNext-152) | BERT | Microsoft-OCR | FastText+PHOC | Decoder | N |
OCR-VQA
| Y-C./J. | Methods | Acc. | I. E. | Q. E. | OCR | O. E. | Output | Ensem. |
|---|---|---|---|---|---|---|---|---|
| 2020--CVPR | M4C | 63.9 | Faster R-CNN | BERT | Rosetta-en | FastText | Decoder | N |
Similar repos
Contributors
27 commits
1 commits