this repo has no description
dekker.one
1MiniZinc Testing
2================
3
4## Setup
5
6Requires Python 3 (on Windows 3.8 is required). Make sure you're in the `tests/` directory.
7
8```sh
9pip install -r requirements.txt
10```
11
12## Running Test Suite
13
14To run the full test suite:
15
16```sh
17pytest
18```
19
20An HTML report will be generated at `output/report.html`.
21
22Use the `--solvers` option to specify a subset of solvers to use in the tests. Tests which use a solver not in this list will be skipped.
23```sh
24pytest --solvers gecode,chuffed
25```
26
27Use the `--driver` option to specify the directory containing the minizinc executable. Otherwise, default directories and PATH will be used to locate it.
28```sh
29pytest --driver=../build
30```
31
32## Multiple test suites
33
34To facilitate running the test suite with different minizinc options, `specs/suites.yml` contains configurations for running tests, or a subset of tests using different options.
35
36```yaml
37my-test-suite: !Suite
38 includes: ['*'] # Globs for included .mzn files
39 solvers: [gecode, chuffed] # The allowed solvers (if the test case itself specifies a different solver it will be skipped)
40 strict: false # Allow tests to pass if they check against another solver (default true)
41 options:
42 -O3: true # Default options to pass to minizinc (merged and overwritten by individual test cases)
43```
44
45For example, to run the `optimize-2` and `no-mip-domains` configurations only:
46
47```sh
48pytest --suite optimize-2 --suite no-mip-domains
49```
50
51## Creating/editing test cases with the web interface
52
53A web interface can be used to create and edit test cases graphically. Python 3.8.1 or newer and Flask is required.
54
55```sh
56pip install flask
57```
58
59Start the web interface with
60
61```sh
62cd tests
63python -m minizinc_testing.create
64```
65
66Then open `http://localhost:5000`.
67
68The web interface detects `.mzn` files in the `tests/spec` directory. To add a new test, simply create a `.mzn` containing the model, and then open the file in the web interface. Test cases can then be generated accordingly.
69
70## Creating/editing test cases manually
71
72The test cases are defined using `YAML` inside the minizinc `.mzn` files. This YAML definition must be inside a block comment like the following:
73
74```c
75/***
76!Test
77expected: !Result
78 solution:
79 x: 1
80***/
81```
82
83Multiple cases can be specified for one `.mzn` file:
84
85```c
86/***
87--- !Test
88 ...
89--- !Test
90 ...
91--- !Test
92 ...
93***/
94```
95
96### YAML format
97
98The format of the test case spec is as follows:
99
100```yaml
101!Test
102solvers: [gecode, cbc, chuffed] # List of solvers to use (omit if all solvers should be tested)
103check_against: [gecode, cbc, chuffed] # List of solvers used to check results (omit if no checking is needed)
104extra_files: [datafile.dzn] # Data files to use if any
105options: # Options passed to minizinc-python's solve(), usually all_solutions if present
106 all_solutions: true
107 timeout: !Duration 10s
108expected: # The obtained result must match one of these
109- !Result
110 status: SATISFIED # Result status
111 solution: !Solution
112 s: 1
113 t: !!set {1, 2, 3} # The set containing 1, 2 and 3
114 u: !Range 1..10 # The range 1 to 10 (inclusive)
115 v: [1, 2, 3] # The array with 1, 2, 3
116 x: !Unordered [3, 2, 1] # Ignore the order of elements in this array
117 _output_item: !Trim |
118 trimmed output item
119 gets leading/trailing
120 whitespace ignored
121- !Error
122 type: MiniZincError # Name of the error type
123 message: Exact error message # Exact error message string (avoid using this as it's generally not portable)
124 regex: .*type-inst must be par set.* # Regex the start of the string must match (run with M and S flags)
125```
126
127For a test to pass, at least one expected result must be a subset of the obtained result. That is, the obtained result can have more attributes, but not less, and corresponding attributes must match.
128
129If a solution is produced that does not match any given expected output, the result is checked using another solver. If this check passes then the test passes with a warning.
130
131### Multiple solutions
132
133When setting `all_solutions: true` and the order of the returned solutions often does not matter, use `!SolutionSet` for the list of solutions:
134
135```yaml
136!Result
137 status: ALL_SOLUTIONS
138 solution: !SolutionSet
139 - !Solution
140 x: 1
141 - !Solution
142 x: 2
143```
144
145### Testing FlatZinc output
146
147Use `type: compile` on a test to enable only flattening.
148Then `!FlatZinc filename.fzn` to give files with expected results.
149
150```yaml
151!Test
152solvers: [gecode]
153type: compile
154expected: !FlatZinc expected.fzn
155```
156
157## TODO
158
159- Tool for generating test cases
160- Tweak YAML parsing so not so many !Tags are required
161- Better documentation of how the framework operates