this repo has no description
hauleth.dev
1+++
2title = "Common Test for Elixir developers"
3date = 2019-07-15
4
5[taxonomies]
6tags = [
7 "erlang",
8 "beam",
9 "elixir",
10 "testing",
11 "programming",
12 "common_test",
13 "commoner"
14]
15+++
16
17In my new job I have opportunity to work a little bit more with Erlang code and
18Erlang related tools than earlier. That forced me a little to learn and use
19[Common Test][ct] library which is **the** testing library in Erlang world.
20When after that I needed to work back it hit me hard how much stuff I am
21missing in Elixir. In this article I will try to present Common Test library
22from the viewpoint of long standing Elixir guy and present how it compares to
23the ExUnit.
24
25## Unit testing vs integration testing
26
27The main difference between these two is their intended usage. In it's core
28ExUnit is intended as a library for unit testing (no surprise there, as it is
29literally in it's name), on the other hand Common Test is more about integration
30testing and looking on the system as a whole. So the discussion there would not
31be fair, as it isn't apples to apples, but for second let's check what makes
32each of them great for their intended uses:
33
34ExUnit for unit testing:
35
36- Simple way to setup and teardown tests via `setup` and `setup_all` macros
37- Built in testing of documentation examples via `doctest`
38- Immutability of test environment (at least that is what you should try to
39 achieve)
40- Very simple way of running different test modules in parallel (tests within
41 single module are always sync)
42- Each test run in separate process, which prevents sharing data between
43 processes via process dictionary
44- Grouping tests into `describe` blocks
45- In overall simplicity of the library as a whole
46
47Common Test for integration testing:
48
49- Persistent test results - logs, results, "generated garbage" is stored between
50 runs in the Common Test, which allows to track what and when happened, and if
51 needed trace the bug in the logs/leftovers
52- Generating "private directory" per suite
53- Grouping tests while allowing single test to be reused in different groups
54- Extensive control about order and parallelization of tests within groups
55- Built in very extensive and detailed reports (HTML and JUnit)
56- Distributed testing
57- Define dependencies between tests
58- Set of helpers for working with different network protocols (SSH, Telnet,
59 SNMP, FTP, Erlang RPC, netconf)
60- Built in support for data fixtures
61- Built in support for complex configuration of test runs via [test
62 specifications](http://erlang.org/doc/apps/common_test/run_test_chapter.html#test-specifications)
63- Built in logging (we will cover that later)
64
65So as you can see, the Common Test is much broader and complex, but at the same
66time it provides a lot of utilities that are really helpful while writing
67integration tests, that can span multiple applications.
68
69## Writing Common Test suite
70
71While CT provides enormous power in hand of the experienced tester/developer
72it's API isn't the top of the class (which happen a lot in Erlang tools, whoever
73wanted to integrate them with anything then they will know what I mean). But in
74fact writing simple test suite in CT is fairly easy:
75
76```erlang
77-module(example_SUITE). % The naming convention (with uppercase _SUITE) Erlang
78 % convention which allow ct to find test suites.
79 % Something like ExUnit _test.exs naming convention
80
81-export([all/0]).
82
83-export([test_function_name/1]).
84
85all() ->
86 [test_function_name].
87
88test_function_name(_Config) ->
89 2 = 1 + 1.
90```
91
92It is pretty easy and understandable format, but with due CT power it [quickly
93can grow to be "less" readable](https://github.com/erlang/otp/blob/81d332cc693d2be8a5af16bfbcae0bfde6702479/lib/ssh/test/ssh_algorithms_SUITE.erl#L36).
94Taming that power can be problematic and can cause few headaches for newcomers.
95
96## Reporting
97
98Another part, and the one that I miss the most in ExUnit, is reporting in Common
99Test. This alone is in my humble opinion **THE** best part of the library. To
100present you how this work assume that we have above test suite which we will run
101with `ct_run -dir .`. This will compile and run our tests (obviously), but in
102addition to that it will generate hell lot of cruft files, some JQuery (yes,
103this is still used in 2019), some CSS files, some HTML and other.
104
105Just check [this out](/common-test-example/simple/index.html). This is example report
106generated by the Common Test. As you can see it contains a lot of information in
107quite readable format. Not only it contains informations about current run, but
108all previous runs as well, which is really handy when tracking regressions.
109
110But can we store even more informations there? Yes, as CT includes simple
111logging facility it is completely possible to log your own informations during
112tests, for example, lets modify our test to log some informations:
113
114```erlang
115test_function_name(_Config) ->
116 ct:log("Example message"),
117 2 = 1 + 1.
118```
119
120Now when we run tests again, then we will see more informations (even coloured)
121in [our test log](/common-test-example/log/ct_run.ct@NiunioBook.2019-07-16_11.03.21/common-test-example.log.logs/run.2019-07-16_11.03.22/example_suite.test_function_name.html):
122
123
124
125Additionally there is support Surefire XML output (commonly known as JUnit XML)
126via [hook that is distributed with Common Test](http://erlang.org/doc/apps/common_test/ct_hooks_chapter.html#built-in-cths).
127This is very useful, as many CI services (for sure Jenkins, Circle CI, and
128GitLab CI) support such files to provide better CI results than raw stdout.
129
130## Comparison to ExUnit
131
132ExUnit is great tool, which is very flexible while being relatively simple.
133Unfortunately that simplicity makes it very lacking when comparing with "more
134established" solutions like Common Test which shines when we "grow out" of the
135ExUnit tests. It provides a lot great tools that really help with finding bugs
136(for example when I was joining the project and I was trying to run tests
137locally I have some issues due to misconfiguration, it was much easier to send
138tarball of the test results instead of sending wall of text captured from
139stdout).
140
141Don't get me wrong, ExUnit is great and powerful, and for sure you should use
142it. It is great evolution over EUnit, but again, it would be worth of having
143built in [Surefire XML output](http://www.erlang.org/doc/man/eunit_surefire.html)
144due to it's popularity and support in many CI services.
145
146## Future
147
148But is everything lost and we need to fall back to the Erlang code (which can be
149problematic for many people) to use Common Test? Not exactly, Comcast (yes, that
150Comcast) [created simple wrapper][ctex] for Elixir (unfortunately seems pretty
151dead to me) and I have started [Commoner][commoner] library that is meant to
152provide API more familiar and easier to use:
153
154```elixir
155defmodule Commoner.ExampleSuite do
156 use CommonTest.Suite
157
158 test "foo bar" do
159 assert (1 + 1) in [2]
160 end
161
162 test "bar baz" do
163 CommonTest.log("Hello")
164 end
165end
166```
167
168I am very interested about your feelings about Common Test usage in Elixir and
169about proposed API for Commoner. You can ping me:
170
171- On [Elixir forum thread about Commoner][forum]
172- On Twitter via `@hauleth`
173
174### Special thanks
175
176- José Valim - for reading it through and reviewing before publishing
177
178[ct]: http://erlang.org/doc/apps/common_test/basics_chapter.html
179[ctex]: https://github.com/Comcast/ctex
180[commoner]: https://github.com/hauleth/commoner
181[forum]: https://elixirforum.com/t/commoner-elixir-wrapper-for-common-test-library/23966?u=hauleth