Workflow

1. Build. We first make preparations for the testing. Build/pull Docker images of DBMSs, get one version of SQLancer, and running scripts.
2. Test. We start the test by using the preparations.

Goals

• Merge the open pull request by addressing the remaining minor comments.
• Create a README.
• Separate the build process, or cache it, i.e., we can have an option to reuse the previously created Docker images.
• Persist SQLancer logs (and DBMS logs, if available). e.g., in logs/YY-MM-DD directories
• Add support for an additional DBMS, building its Docker image via a Dockerfile rather than pulling it directly.

Examples

Perhaps some user stories could help you with a better solution. Considering below scenarios:

DBMS Developer: test the latest version

DBMS developers want to test their system using SQLancer. They have some customized/latest version of the DBMS in a Dockerfile. Now a test.py (or a start.py that calls build.py and test.py) can 1) build the DBMS by the specified Dockerfile, 2) start SQLancer and DBMS containers with some configurations to test, and 3) check the logs.

The script may be something like (argument names just for demo)

test --dockerfile FILE --config config.json --no-cache 

Testing researcher: test SQLancer

Some researchers may want to add some features to SQLancer and test the latest version of one DBMS. Now the scripts may be like 1) pull the latest DBMS image, and 2) start SQLancer with some modified configurations.

test --dbms DBMS --config config.json --no-cache

It can also be some routine testing; they want to test the same version of the DBMS, and now there's no need to pull/build Docker image.

test --dbms DBMS --config config.json --cache

Hobbyist

Someone is really interested in the reliability of the DBMSs. We want to test all of the DBMSs, e.g., in 5 mins.

test --dbms all