this repo has no description
hauleth.dev
1+++
2title = "Common Test for Elixir developers"
3date = 2019-07-15
4
5[taxonomies]
6tags = [
7 "erlang",
8 "beam",
9 "elixir",
10 "testing",
11 "programming",
12 "common_test",
13 "commoner"
14]
15
16[[extra.thanks]]
17name = "José Valim"
18why = "for reading it through and reviewing before publishing"
19+++
20
21In my new job I have opportunity to work a little bit more with Erlang code and
22Erlang related tools than earlier. That forced me a little to learn and use
23[Common Test][ct] library which is **the** testing library in Erlang world.
24When after that I needed to work back it hit me hard how much stuff I am
25missing in Elixir. In this article I will try to present Common Test library
26from the viewpoint of long standing Elixir guy and present how it compares to
27the ExUnit.
28
29## Unit testing vs integration testing
30
31The main difference between these two is their intended usage. In it's core
32ExUnit is intended as a library for unit testing (no surprise there, as it is
33literally in it's name), on the other hand Common Test is more about integration
34testing and looking on the system as a whole. So the discussion there would not
35be fair, as it isn't apples to apples, but for second let's check what makes
36each of them great for their intended uses:
37
38ExUnit for unit testing:
39
40- Simple way to setup and teardown tests via `setup` and `setup_all` macros
41- Built in testing of documentation examples via `doctest`
42- Immutability of test environment (at least that is what you should try to
43 achieve)
44- Very simple way of running different test modules in parallel (tests within
45 single module are always sync)
46- Each test run in separate process, which prevents sharing data between
47 processes via process dictionary
48- Grouping tests into `describe` blocks
49- In overall simplicity of the library as a whole
50
51Common Test for integration testing:
52
53- Persistent test results - logs, results, "generated garbage" is stored between
54 runs in the Common Test, which allows to track what and when happened, and if
55 needed trace the bug in the logs/leftovers
56- Generating "private directory" per suite
57- Grouping tests while allowing single test to be reused in different groups
58- Extensive control about order and parallelization of tests within groups
59- Built in very extensive and detailed reports (HTML and JUnit)
60- Distributed testing
61- Define dependencies between tests
62- Set of helpers for working with different network protocols (SSH, Telnet,
63 SNMP, FTP, Erlang RPC, netconf)
64- Built in support for data fixtures
65- Built in support for complex configuration of test runs via [test
66 specifications](http://erlang.org/doc/apps/common_test/run_test_chapter.html#test-specifications)
67- Built in logging (we will cover that later)
68
69So as you can see, the Common Test is much broader and complex, but at the same
70time it provides a lot of utilities that are really helpful while writing
71integration tests, that can span multiple applications.
72
73## Writing Common Test suite
74
75While CT provides enormous power in hand of the experienced tester/developer
76it's API isn't the top of the class (which happen a lot in Erlang tools, whoever
77wanted to integrate them with anything then they will know what I mean). But in
78fact writing simple test suite in CT is fairly easy:
79
80```erlang
81-module(example_SUITE). % The naming convention (with uppercase _SUITE) Erlang
82 % convention which allow ct to find test suites.
83 % Something like ExUnit _test.exs naming convention
84
85-export([all/0]).
86
87-export([test_function_name/1]).
88
89all() ->
90 [test_function_name].
91
92test_function_name(_Config) ->
93 2 = 1 + 1.
94```
95
96It is pretty easy and understandable format, but with due CT power it [quickly
97can grow to be "less" readable](https://github.com/erlang/otp/blob/81d332cc693d2be8a5af16bfbcae0bfde6702479/lib/ssh/test/ssh_algorithms_SUITE.erl#L36).
98Taming that power can be problematic and can cause few headaches for newcomers.
99
100## Reporting
101
102Another part, and the one that I miss the most in ExUnit, is reporting in Common
103Test. This alone is in my humble opinion **THE** best part of the library. To
104present you how this work assume that we have above test suite which we will run
105with `ct_run -dir .`. This will compile and run our tests (obviously), but in
106addition to that it will generate hell lot of cruft files, some JQuery (yes,
107this is still used in 2019), some CSS files, some HTML and other.
108
109Just check [this out](/common-test-example/simple/index.html). This is example report
110generated by the Common Test. As you can see it contains a lot of information in
111quite readable format. Not only it contains informations about current run, but
112all previous runs as well, which is really handy when tracking regressions.
113
114But can we store even more informations there? Yes, as CT includes simple
115logging facility it is completely possible to log your own informations during
116tests, for example, lets modify our test to log some informations:
117
118```erlang
119test_function_name(_Config) ->
120 ct:log("Example message"),
121 2 = 1 + 1.
122```
123
124Now when we run tests again, then we will see more informations (even coloured)
125in [our test log](/common-test-example/log/ct_run.ct@NiunioBook.2019-07-16_11.03.21/common-test-example.log.logs/run.2019-07-16_11.03.22/example_suite.test_function_name.html):
126
127
128
129Additionally there is support Surefire XML output (commonly known as JUnit XML)
130via [hook that is distributed with Common Test](http://erlang.org/doc/apps/common_test/ct_hooks_chapter.html#built-in-cths).
131This is very useful, as many CI services (for sure Jenkins, Circle CI, and
132GitLab CI) support such files to provide better CI results than raw stdout.
133
134## Comparison to ExUnit
135
136ExUnit is great tool, which is very flexible while being relatively simple.
137Unfortunately that simplicity makes it very lacking when comparing with "more
138established" solutions like Common Test which shines when we "grow out" of the
139ExUnit tests. It provides a lot great tools that really help with finding bugs
140(for example when I was joining the project and I was trying to run tests
141locally I have some issues due to misconfiguration, it was much easier to send
142tarball of the test results instead of sending wall of text captured from
143stdout).
144
145Don't get me wrong, ExUnit is great and powerful, and for sure you should use
146it. It is great evolution over EUnit, but again, it would be worth of having
147built in [Surefire XML output](http://www.erlang.org/doc/man/eunit_surefire.html)
148due to it's popularity and support in many CI services.
149
150## Future
151
152But is everything lost and we need to fall back to the Erlang code (which can be
153problematic for many people) to use Common Test? Not exactly, Comcast (yes, that
154Comcast) [created simple wrapper][ctex] for Elixir (unfortunately seems pretty
155dead to me) and I have started [Commoner][commoner] library that is meant to
156provide API more familiar and easier to use:
157
158```elixir
159defmodule Commoner.ExampleSuite do
160 use CommonTest.Suite
161
162 test "foo bar" do
163 assert (1 + 1) in [2]
164 end
165
166 test "bar baz" do
167 CommonTest.log("Hello")
168 end
169end
170```
171
172I am very interested about your feelings about Common Test usage in Elixir and
173about proposed API for Commoner. You can ping me:
174
175- On [Elixir forum thread about Commoner][forum]
176- On Twitter via `@hauleth`
177
178[ct]: http://erlang.org/doc/apps/common_test/basics_chapter.html
179[ctex]: https://github.com/Comcast/ctex
180[commoner]: https://github.com/hauleth/commoner
181[forum]: https://elixirforum.com/t/commoner-elixir-wrapper-for-common-test-library/23966?u=hauleth