Awesome-Database-Testing

This is a curitated list of resources on database testing.

What covered: papers, blogs, notes, tools and etc. for whoever wants to test a database mangement system (DBMS).

What is not: using SQLs, learning database basic knowledge, implementing a database system.

Note: this is NOT an exhaustive list of materials, you may want to find more accroding to references of each item.

High-level Views

Here we put materials with general discussions.

Research Papers

NOTE: We only include peer-reviewed and published papers here, the same also applies to other entries. You may want to find the authors' free-version of the papers on their personal page.

• Understanding the query optimization Query Optimization in Database Systems [1984]

• Understanding the query optimization - a more recent view An Overview of Query Optimization [1998]

• What are the problems from the indutrial view Testing SQL Server's Query Optimizer : Challenges , Techniques and Experiences

• Optimizer Evaluation How good are query optimizers, really?

Others

• What are the problems from the academic view Is Query Optimization a “Solved” Problem?

Input Generation

Research Papers

• Query generation with template substitution Generating Thousand Benchmark Queries in Seconds
• Language for customized data generator Flexible Database Generators
• RAGS Microsoft SQL Server Massive Stochastic Testing of SQL
• Generate data regarding the query constraints QAGen: Generating Query-Aware Test Databases
• How could symbolic execution help Qex: Symbolic SQL Query Explorer
• Data generation as a search problem Search-based test data generation for SQL queries
• For more valid inputs! SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback [CCS 2021]

Tools

• Randgen MySQL version (not maintained) doc, MariaDB version, PingCAP version
• pquery - stress testing MySQL
• Sysbench - MySQL benchmarks
• SQLsmith - fuzzing style tool
• Squirrel - tool for CCS 2021 paper
• mtr_to_sql.sh@Percona-QA - a script to extract mysql-test cases as a sql file, with engine replacement and shuffles according to your need

Engine Correctness

Here we mainly refer to the case where the engine returns wrong query result.

Research Papers

What You Really Want to Know About

• RAGS with system-differential testing Massive Stochastic Testing of SQL
• Data generation with contraints sovling Query-Aware Test Generation Using a Relational Constraint Solver
• Differential testing for the optimizing rules A framework for testing query transformation rules
• Criteria on integrtity testing The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints

What You Want to Know About

• Parallelism on the old machines Quickly generating billion-record synthetic databases
• Verified database system Toward a Verified Relational Database Management System
• Reply the transactions Debugging transactions and tracking their provenance with reenactment

What's New

• Search-based test data generation for SQL queries [2018]
• Automated verification of query equivalence using satisfiability modulo theories [2019]
• Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction [2020]
• Testing query execution engines with mutations [2020]

Tools

• SQLancer@ETH_ZURICH
• go-sqlancer@PingCAP
• pquery@Percona-QA - a tool providing the ability to combine correctness and crash recovery testing on both single-node and clustered instances

Performance

Here we focus on detecting and debugging performance issues.

Research Papers

What You Really Want to Know About

• Very first system Efficient testing of high performance transaction processing systems
• Is the exact estimation possible Exact Cardinality Query Optimization for Optimizer Testing
• How to identify issues on plan generation On the stability of plan costs and the costs of plan stability
• What's wrong with the models Testing cardinality estimation models in SQL server

What You Want to Know About

• Generate and compare the plans Testing the accuracy of query optimizers

What's New

• APOLLO: Automatic Detection and Diagnosis of Performance Regressions in Database Systems [2020]

Tools

• TPC Benchmarking
• APOLLO@gatech

Industry Practice

We list idea and tools which are adopted in the industry.

Tools

• Chaos Mesh from PingCAP
  • Article: Building an Automated Testing Framework Based on Chaos Mesh® and Argo
  • Talk: Testing TiDB Using Chaos Mesh® with TiPocket

Contributors

• zhangysh1995
• mkx22

Contributions are Super Welcome!

What you can contribute:

• papers (only peer-reviewed papers will be accepted to this repo)
• talks/slides/articles
• tools

Please also add yourself to Contributors in the PR.