My agentic slop goes here. Not intended for anyone else!
anil.recoil.org
1expect-test - a Cram-like framework for OCaml
2=============================================
3
4
5# Introduction
6
7Expect-test is a framework for writing tests in OCaml, similar to
8[Cram](https://bitheap.org/cram/).
9
10Expect-tests mimic the (now less idiomatic)
11[inline test](https://github.com/janestreet/ppx_inline_test)
12framework in providing a
13`let%expect_test` construct.
14
15The body of an expect-test can contain output-generating code, interleaved with
16`[%expect]` extension expressions to denote the expected output.
17
18When run, expect-tests pass iff the output [_matches_](#matching-behavior) the expected
19output. If a test fails, the `inline_tests_runner` outputs a diff and creates a file with
20the suffix ".corrected" containing the actual output.
21
22Here is an example expect-test in `foo.ml`:
23
24<!-- $MDX file=./test/negative-tests/for-mdx/foo.ml,part=addition -->
25```ocaml
26open! Core
27
28let%expect_test "addition" =
29 printf "%d" (1 + 2);
30 [%expect {| 4 |}]
31;;
32```
33
34When the test runs, the `inline_tests_runner` creates `foo.ml.corrected` with contents:
35
36<!-- $MDX file=./test/negative-tests/for-mdx/foo.ml.corrected.expected,part=addition -->
37```ocaml
38open! Core
39
40let%expect_test "addition" =
41 printf "%d" (1 + 2);
42 [%expect {| 3 |}]
43;;
44```
45
46`inline_tests_runner` also outputs:
47
48<!-- $MDX file=./test/negative-tests/for-mdx/test-output.expected -->
49```
50------ foo.ml
51++++++ foo.ml.corrected
52File "foo.ml", line 6, characters 0-1:
53 |open! Core
54 |
55 |let%expect_test "addition" =
56 | printf "%d" (1 + 2);
57-| [%expect {| 4 |}]
58+| [%expect {| 3 |}]
59 |;;
60 |
61```
62
63Diffs are shown in color if the `-use-color` flag is passed to the inline test runner
64executable.
65
66# Common usage
67
68Each `[%expect]` block matches all the output generated since the previous `[%expect]`
69block (or the beginning of the test). In this way, when multiple `[%expect]` blocks are
70interleaved with test code, they can help show which part of the test produced which
71output.
72
73The following test:
74
75<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=interleaved -->
76```ocaml
77let%expect_test "interleaved" =
78 let l = [ "a"; "b"; "c" ] in
79 printf "A list [l]\n";
80 printf "It has length %d\n" (List.length l);
81 [%expect {| A list [l] |}];
82 List.iter l ~f:print_string;
83 [%expect
84 {|
85 It has length 3
86 abc
87 |}]
88;;
89```
90
91is rewritten as
92
93<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=interleaved -->
94```ocaml
95let%expect_test "interleaved" =
96 let l = [ "a"; "b"; "c" ] in
97 printf "A list [l]\n";
98 printf "It has length %d\n" (List.length l);
99 [%expect
100 {|
101 A list [l]
102 It has length 3
103 |}];
104 List.iter l ~f:print_string;
105 [%expect {| abc |}]
106;;
107```
108
109When there is "trailing" output at the end of a `let%expect_test` (output that has yet to
110be matched by some `[%expect]` block), a new `[%expect]` block is appended to the test
111with the trailing output:
112
113<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=trailing -->
114```ocaml
115let%expect_test "trailing output" =
116 print_endline "Hello";
117 [%expect {| Hello |}];
118 print_endline "world"
119;;
120```
121
122becomes:
123
124<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=trailing -->
125```ocaml
126let%expect_test "trailing output" =
127 print_endline "Hello";
128 [%expect {| Hello |}];
129 print_endline "world";
130 [%expect {| world |}]
131;;
132```
133
134# Matching behavior
135
136You might have noticed that the contents of the `[%expect]` blocks are not _exactly_ the
137program output; in some of the examples above, they contain a different number of leading
138and trailing newlines, and are indented to match the code indentation. We say the contents
139of a block `[%expect str]` (where `str` is some string literal) _match_ the output at that
140block if the output, after we format it to standardize indentation and other whitespace,
141is identical to the contents of `str`
142after it has been similarly formatted
143.
144
145The formatting applied depends on the type of delimiter used in `str` (i.e. whether it a
146`"quoted string"` or a `{xxx| delimited string |xxx}`). To summarize:
147
148* Output containing only whitespace is formatted as `[%expect {| |}]` or `[%expect ""]`.
149* Output where only one line contains non-whitespace characters is formatted onto a single
150 line, as `[%expect {| output |}]` or `[%expect "output"]`.
151* Output where multiple lines contain non-whitespace characters is formatted so that:
152 - There is no trailing whitespace on lines with content.
153 - The relative indentation of the lines is preserved.
154 - In `{| delimited strings |}`, the least-indented line with content (the "left margin"
155 of the output) is aligned to be two spaces past the indentation of the `[%expect]`
156 block.
157 - In `"quoted string"`, the least-indented line is indented by exactly one space (this
158 plays the nicest with `ocamlformat`'s existing decisions about how to format string
159 literals).
160 - There is one empty line before and one empty line after the contents.
161
162
163Here is an example containing several cases of output that are subject to distinct
164formatting rules and how they appear in `[%expect]` and `[%expect_exact]` blocks:
165
166<details>
167<summary>Expand examples</summary>
168
169<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=matching -->
170```ocaml
171let%expect_test "matching behavior --- no content" =
172 printf " ";
173 [%expect {| |}];
174 printf " ";
175 [%expect ""];
176 printf " ";
177 [%expect_exact {| |}];
178 printf " ";
179 [%expect_exact " "]
180;;
181
182let%expect_test "matching behavior --- one line of content" =
183 printf "\n This is one line\n\n";
184 [%expect {| This is one line |}];
185 printf "\n This is one line\n\n";
186 [%expect "This is one line"];
187 printf "\n This is one line\n\n";
188 [%expect_exact
189 {|
190 This is one line
191
192|}];
193 printf "\n This is one line\n\n";
194 [%expect_exact "\n This is one line\n\n"]
195;;
196
197let%expect_test "matching behavior --- multiple lines of content" =
198 printf
199 {|
200Once upon a midnight dreary,
201 while I pondered, weak and weary,
202Over many a quaint and curious
203 volume of forgotten lore |};
204 [%expect
205 {|
206 Once upon a midnight dreary,
207 while I pondered, weak and weary,
208 Over many a quaint and curious
209 volume of forgotten lore
210 |}];
211 printf
212 {|
213Once upon a midnight dreary,
214 while I pondered, weak and weary,
215Over many a quaint and curious
216 volume of forgotten lore |};
217 [%expect
218 " \n\
219 \ Once upon a midnight dreary,\n\
220 \ while I pondered, weak and weary,\n\
221 \ Over many a quaint and curious\n\
222 \ volume of forgotten lore\n\
223 \ "];
224 printf
225 {|
226Once upon a midnight dreary,
227 while I pondered, weak and weary,
228Over many a quaint and curious
229 volume of forgotten lore |};
230 [%expect_exact
231 {|
232Once upon a midnight dreary,
233 while I pondered, weak and weary,
234Over many a quaint and curious
235 volume of forgotten lore |}];
236 printf
237 {|
238Once upon a midnight dreary,
239 while I pondered, weak and weary,
240Over many a quaint and curious
241 volume of forgotten lore |};
242 [%expect_exact
243 "\n\
244 Once upon a midnight dreary,\n\
245 \ while I pondered, weak and weary,\n\
246 Over many a quaint and curious\n\
247 \ volume of forgotten lore "]
248;;
249```
250</details>
251
252Expect-test is by default permissive about this formatting, so that a
253`[%expect]` block that is correct modulo formatting is
254accepted. However, passing `-expect-test-strict-indentation=true` to
255the ppx driver makes the test runner issue corrections for blocks that
256do not satisfy the indentation rules.
257For example, the following:
258
259<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=bad-format -->
260```ocaml
261let%expect_test "bad formatting" =
262 printf "a\n b";
263 [%expect
264 {|
265a
266 b |}]
267;;
268```
269is corrected to:
270
271<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=bad-format -->
272```ocaml
273let%expect_test "bad formatting" =
274 printf "a\n b";
275 [%expect
276 {|
277 a
278 b
279 |}]
280;;
281```
282
283(to add the required indentation and trailing newline)
284
285
286# Reachability
287
288## Expects reached from multiple places
289
290A `[%expect]` extension can be encountered multiple times if it is in e.g. a functor or a
291function:
292
293<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=function -->
294```ocaml
295let%expect_test "function" =
296 let f output =
297 print_string output;
298 [%expect {| hello world |}]
299 in
300 f "hello world";
301 f "hello world"
302;;
303```
304
305The test passes if the `[%expect]` block matches the output each time it is encountered,
306as described in the section on [matching behavior](#matching-behavior).
307
308If the outputs are not consistent, then the corrected file contains a report of all of the
309outputs that were captured, in the order that they were captured at runtime.
310
311For example, calling `f` in the snippet above with inconsistent arguments will produce:
312
313<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=broken-function -->
314```ocaml
315let%expect_test "function" =
316 let f output =
317 print_string output;
318 [%expect
319 {|
320 (* expect_test: Test ran multiple times with different test outputs *)
321 ============================ Output 1 / 4 ============================
322 hello world
323 ============================ Output 2 / 4 ============================
324 goodbye world
325 ============================ Output 3 / 4 ============================
326 once upon
327 a midnight dreary
328 ============================ Output 4 / 4 ============================
329 hello world
330 |}]
331 in
332 f "hello world";
333 f "goodbye world";
334 f "once upon\na midnight dreary";
335 f "hello world"
336;;
337```
338
339
340## Unreached expects
341
342Every `[%expect]` and `[%expect_exact]` block in a `let%expect_test` must be reached at
343least once if that test is ever run. Failure for control flow to reach a block is _not_
344treated like recording empty output at a block. The extension expression
345`[%expect.unreachable]` is used to indicate that some part of an expect test shouldn't be
346reached; if control flow reaches that point anyway, the corrected file replaces the
347`[%expect.unreachable]` with a plain old expect containing the output collected until that
348point. Conversely, if control flow never reaches some `[%expect]` block, that block is
349turned into a `[%expect.unreachable]`. For example:
350
351<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=unreachable -->
352```ocaml
353let%expect_test "unreachable" =
354 let interesting_bool = 3 > 5 in
355 printf "%b\n" interesting_bool;
356 if interesting_bool
357 then [%expect {| true |}]
358 else (
359 printf "don't reach\n";
360 [%expect.unreachable])
361;;
362```
363becomes:
364
365<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=unreachable -->
366```ocaml
367let%expect_test "unreachable" =
368 let interesting_bool = 3 > 5 in
369 printf "%b\n" interesting_bool;
370 if interesting_bool
371 then [%expect.unreachable]
372 else (
373 printf "don't reach\n";
374 [%expect
375 {|
376 false
377 don't reach
378 |}])
379;;
380```
381
382Note that, for an expect block that is sometimes reachable and sometimes not, that block
383passes if the output captured at that block matches every time the block is encountered.
384For example, the following test passes:
385
386<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sometimes-reachable -->
387```ocaml
388module Test (B : sig
389 val interesting_opt : int option
390 end) =
391struct
392 let%expect_test "sometimes reachable" =
393 match B.interesting_opt with
394 | Some x ->
395 printf "%d\n" x;
396 [%expect {| 5 |}]
397 | None -> [%expect {| |}]
398 ;;
399end
400
401module _ = Test (struct
402 let interesting_opt = Some 5
403 end)
404
405module _ = Test (struct
406 let interesting_opt = None
407 end)
408
409module _ = Test (struct
410 let interesting_opt = Some 5
411 end)
412```
413
414# Exceptions
415
416When an exception is raised by the body of an expect-test, the `inline_test_runner` shows
417it (and, if relevant, any output generated by the test that had not yet been captured) in
418a `[@@expect.uncaught_exn]` attribute attached to the corresponding `let%expect_test`.
419`[%expect]` blocks in the test are treated according to the usual rules: those reached
420before the exception is raised capture output as usual, and those that "would have" been
421reached after are marked as unreachable:
422
423<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml,part=exn -->
424```ocaml
425let%expect_test "exception" =
426 Printexc.record_backtrace false;
427 printf "start!";
428 [%expect {| |}];
429 let sum = 2 + 2 in
430 if sum <> 3
431 then (
432 printf "%d" sum;
433 failwith "nope");
434 printf "done!";
435 [%expect {| done! |}]
436;;
437```
438
439becomes:
440
441<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=exn -->
442```ocaml
443let%expect_test "exception" =
444 Printexc.record_backtrace false;
445 printf "start!";
446 [%expect {| start! |}];
447 let sum = 2 + 2 in
448 if sum <> 3
449 then (
450 printf "%d" sum;
451 failwith "nope");
452 printf "done!";
453 [%expect.unreachable]
454[@@expect.uncaught_exn
455 {|
456 (Failure nope)
457 Trailing output
458 ---------------
459 4
460 |}]
461;;
462```
463
464Unlike `[%expect]` blocks, which might be reached on some runs of a test and not others, a
465test with an `[@@expect.uncaught_exn]` attribute _must_ raise every time it is run.
466Changing the `None` branch of the functorized test from [before](#unreached-expects) to
467raise gives:
468
469<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sometimes-raises -->
470```ocaml
471module Test' (B : sig
472 val interesting_opt : int option
473 end) =
474struct
475 let%expect_test "sometimes raises" =
476 match B.interesting_opt with
477 | Some x ->
478 printf "%d\n" x;
479 [%expect {| 5 |}]
480 | None -> failwith "got none!"
481 [@@expect.uncaught_exn
482 {|
483 (* expect_test: Test ran multiple times with different uncaught exceptions *)
484 =============================== Output 1 / 3 ================================
485 <expect test ran without uncaught exception>
486 =============================== Output 2 / 3 ================================
487 (Failure "got none!")
488 =============================== Output 3 / 3 ================================
489 <expect test ran without uncaught exception>
490 |}]
491 ;;
492end
493
494module _ = Test' (struct
495 let interesting_opt = Some 5
496 end)
497
498module _ = Test' (struct
499 let interesting_opt = None
500 end)
501
502module _ = Test' (struct
503 let interesting_opt = Some 5
504 end)
505```
506
507
508# Output capture
509
510The extension point `[%expect.output]` evaluates to a `string` with the output that would
511have been captured had an `[%expect]` node been there instead.
512
513One idiom for testing non-deterministic output is to capture the output using
514`[%expect.output]` and post-process it:
515
516<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=output-capture -->
517```ocaml
518(* Suppose we want to test code that attaches a timestamp to everything it prints *)
519let print_message s = printf "%s: %s\n" (Time_float.to_string_utc (Time_float.now ())) s
520
521let%expect_test "output capture" =
522 (* A simple way to clean up the non-determinism is to 'X' all digits *)
523 let censor_digits s = String.map s ~f:(fun c -> if Char.is_digit c then 'X' else c) in
524 print_message "Hello";
525 [%expect.output] |> censor_digits |> print_endline;
526 [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];
527 print_message "world";
528 [%expect.output] |> censor_digits |> print_endline;
529 [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]
530;;
531```
532
533Other uses of `[%expect.output]` include:
534
535* Sorting lines of output printed in nondeterministic order.
536* Passing output that is known to be a sexp to `t_of_sexp` and performing tests on the
537 resulting structure.
538* Performing some sort of additional validation on the output before printing it to a
539 normal `[%expect]` block.
540
541# Configuration
542
543Expect-test exposes hooks for configuring how the bodies of expect tests are run, which
544can be used to set up and tear down test environments, sanitize output, or embed
545`[%expect]` expressions in a monadic computation, like a `Deferred.t`.
546
547Each `let%expect_test` reads these configurations from the module named
548`Expect_test_config` in the scope of that let binding. The default module in scope defines
549no-op hooks that the user can override. To do so, first include the existing
550`Expect_test_config`, then override a subset of the following interface:
551
552```ocaml
553module type Expect_test_config = sig
554 (** The type of the expression on the RHS of a [let%expect_test]
555 binding is [unit IO.t] *)
556 module IO : sig
557 type 'a t
558
559 val return : 'a -> 'a t
560 end
561
562 (** Run an IO operation until completion *)
563 val run : (unit -> unit IO.t) -> unit
564
565 (** [sanitize] can be used to map all output strings, e.g. for cleansing. *)
566 val sanitize : string -> string
567
568 (** This module type actually contains other definitions, but they
569 are for internal testing of [ppx_expect] only. *)
570end
571```
572
573For example, `Async` exports an `Expect_test_config` equivalent to:
574
575```ocaml skip
576module Expect_test_config = struct
577 include Expect_test_config
578
579 module IO = Async_kernel.Deferred
580
581 let run f = Async_unix.Thread_safe.block_on_async_exn f
582end
583```
584
585If we want to consistently apply the same sanitization to all of the output in our expect
586test, like we did in the timestamp example above, we can override
587`Expect_test_config.sanitize`. This cleans up the testing code and removes the need to use
588`[%expect.output]`.
589
590<!-- $MDX file=./test/negative-tests/for-mdx/mdx_cases.ml.corrected.expected,part=sanitization -->
591```ocaml
592(* Suppose we want to test code that attaches a timestamp to everything it prints *)
593let print_message s = printf "%s: %s\n" (Time_float.to_string_utc (Time_float.now ())) s
594
595module Expect_test_config = struct
596 include Expect_test_config
597
598 (* A simple way to clean up the non-determinism is to 'X' all digits *)
599 let sanitize s = String.map s ~f:(fun c -> if Char.is_digit c then 'X' else c)
600end
601
602let%expect_test "sanitization" =
603 print_message "Hello";
604 [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: Hello |}];
605 print_message "world";
606 [%expect {| XXXX-XX-XX XX:XX:XX.XXXXXXZ: world |}]
607;;
608```
609
610# Debugging deadlocks & hanging tests
611
612
613One common pitfall while using expect-test is that it hurts the inspectability of code
614that fails to terminate. Normally, adding prints to understand how code behaves interacts
615well with expect-test; when the tests finish, the test framework produces a corrected
616file, where the output has been inserted to the interleaved `[%expect]` extension points.
617However, if a test never finishes, no corrected file is produced at all, and all of the
618useful debug output produced by the program seemingly vanishes into oblivion.
619
620As of December 2024, it is possible to see the output collected by a running expect-test
621program in real-time using the `-verbose` flag[^1].
622
623For example, the output of the following OCaml program (`test_loops.ml`)
624
625```ocaml skip
626let rec loop () = loop ()
627
628let%expect_test "doesn't finish" =
629 print_endline "about to enter an infinite loop";
630 loop ()
631;;
632```
633
634can be seen by running
635
636
637```
638$ ./inline_tests_runner -verbose -only-test test_loops.ml
639File "test_loops.ml", line X, characters X-X: doesn't finish
640about to enter an infinite loop
641```
642
643despite the fact that the test never finishes or produces a corrected file.
644
645When `-verbose` is used, running the `inline_tests_runner` executable to completion
646still produces a corrected file, so the `-verbose` flag need not be incompatible with
647normal workflows.
648
649
650
651# Build system integration
652
653Follow the same rules as for
654[ppx_inline_test](https://github.com/janestreet/ppx_inline_test?tab=readme-ov-file#building-and-running-the-tests-outside-of-jane-street-with-dune).
655
656
657[^1]: This behavior is only supported on `inline_tests_runner` executables built for
658 native code on Linux systems. It is not supported on Windows or `inline_tests_runner`s
659 compiled to WASM or javascript.