this repo has no description
hauleth.dev
1---
2title: "Common Test for Elixir developers"
3date: 2019-07-15T14:31:21+02:00
4description: "What is Common Test and how we can use it in Elixir"
5tags:
6 - erlang
7 - beam
8 - elixir
9 - testing
10 - programming
11 - common_test
12 - commoner
13---
14
15In my new job I have opportunity to work a little bit more with Erlang code and
16Erlang related tools than earlier. That forced me a little to learn and use
17[Common Test][ct] library which is **the** testing library in Erlang world.
18When after that I needed to work back it hit me hard how much stuff I am
19missing in Elixir. In this article I will try to present Common Test library
20from the viewpoint of long standing Elixir guy and present how it compares to
21the ExUnit.
22
23## Unit testing vs integration testing
24
25The main difference between these two is their intended usage. In it's core
26ExUnit is intended as a library for unit testing (no surprise there, as it is
27literally in it's name), on the other hand Common Test is more about integration
28testing and looking on the system as a whole. So the discussion there would not
29be fair, as it isn't apples to apples, but for second let's check what makes
30each of them great for their intended uses:
31
32ExUnit for unit testing:
33
34- Simple way to setup and teardown tests via `setup` and `setup_all` macros
35- Built in testing of documentation examples via `doctest`
36- Immutability of test environment (at least that is what you should try to
37 achieve)
38- Very simple way of running different test modules in parallel (tests within
39 single module are always sync)
40- Each test run in separate process, which prevents sharing data between
41 processes via process dictionary
42- Grouping tests into `describe` blocks
43- In overall simplicity of the library as a whole
44
45Common Test for integration testing:
46
47- Persistent test results - logs, results, "generated garbage" is stored between
48 runs in the Common Test, which allows to track what and when happened, and if
49 needed trace the bug in the logs/leftovers
50- Generating "private directory" per suite
51- Grouping tests while allowing single test to be reused in different groups
52- Extensive control about order and parallelization of tests within groups
53- Built in very extensive and detailed reports (HTML and JUnit)
54- Distributed testing
55- Define dependencies between tests
56- Set of helpers for working with different network protocols (SSH, Telnet,
57 SNMP, FTP, Erlang RPC, netconf)
58- Built in support for data fixtures
59- Built in support for complex configuration of test runs via [test
60 specifications](http://www.erlang.org/doc/man/run_test_chapter.html#test-specification-syntax)
61- Built in logging (we will cover that later)
62
63So as you can see, the Common Test is much broader and complex, but at the same
64time it provides a lot of utilities that are really helpful while writing
65integration tests, that can span multiple applications.
66
67## Writing Common Test suite
68
69While CT provides enormous power in hand of the experienced tester/developer
70it's API isn't the top of the class (which happen a lot in Erlang tools, whoever
71wanted to integrate them with anything then they will know what I mean). But in
72fact writing simple test suite in CT is fairly easy:
73
74```erlang
75-module(example_SUITE). % The naming convention (with uppercase _SUITE) Erlang
76 % convention which allow ct to find test suites.
77 % Something like ExUnit _test.exs naming convention
78
79-export([all/0]).
80
81-export([test_function_name/1]).
82
83all() ->
84 [test_function_name].
85
86test_function_name(_Config) ->
87 2 = 1 + 1.
88```
89
90It is pretty easy and understandable format, but with due CT power it [quickly
91can grow to be "less" readable](https://github.com/erlang/otp/blob/81d332cc693d2be8a5af16bfbcae0bfde6702479/lib/ssh/test/ssh_algorithms_SUITE.erl#L36).
92Taming that power can be problematic and can cause few headaches for newcomers.
93
94## Reporting
95
96Another part, and the one that I miss the most in ExUnit, is reporting in Common
97Test. This alone is in my humble opinion **THE** best part of the library. To
98present you how this work assume that we have above test suite which we will run
99with `ct_run -dir .`. This will compile and run our tests (obviously), but in
100addition to that it will generate hell lot of cruft files, some JQuery (yes,
101this is still used in 2019), some CSS files, some HTML and other.
102
103Just check [this out](/common-test-example/simple/index.html). This is example report
104generated by the Common Test. As you can see it contains a lot of information in
105quite readable format. Not only it contains informations about current run, but
106all previous runs as well, which is really handy when tracking regressions.
107
108But can we store even more informations there? Yes, as CT includes simple
109logging facility it is completely possible to log your own informations during
110tests, for example, lets modify our test to log some informations:
111
112```erlang
113test_function_name(_Config) ->
114 ct:log("Example message"),
115 2 = 1 + 1.
116```
117
118Now when we run tests again, then we will see more informations (even coloured)
119in [our test log](/common-test-example/log/ct_run.ct@NiunioBook.2019-07-16_11.03.21/common-test-example.log.logs/run.2019-07-16_11.03.22/example_suite.test_function_name.html):
120
121
122
123Additionally there is support Surefire XML output (commonly known as JUnit XML)
124via [hook that is distributed with Common Test](http://www.erlang.org/doc/man/ct_hooks_chapter.html#built-in-cths).
125This is very useful, as many CI services (for sure Jenkins, Circle CI, and
126GitLab CI) support such files to provide better CI results than raw stdout.
127
128## Comparison to ExUnit
129
130ExUnit is great tool, which is very flexible while being relatively simple.
131Unfortunately that simplicity makes it very lacking when comparing with "more
132established" solutions like Common Test which shines when we "grow out" of the
133ExUnit tests. It provides a lot great tools that really help with finding bugs
134(for example when I was joining the project and I was trying to run tests
135locally I have some issues due to misconfiguration, it was much easier to send
136tarball of the test results instead of sending wall of text captured from
137stdout).
138
139Don't get me wrong, ExUnit is great and powerful, and for sure you should use
140it. It is great evolution over EUnit, but again, it would be worth of having
141built in [Surefire XML output](http://www.erlang.org/doc/man/eunit_surefire.html)
142due to it's popularity and support in many CI services.
143
144## Future
145
146But is everything lost and we need to fall back to the Erlang code (which can be
147problematic for many people) to use Common Test? Not exactly, Comcast (yes, that
148Comcast) [created simple wrapper][ctex] for Elixir (unfortunately seems pretty
149dead to me) and I have started [Commoner][commoner] library that is meant to
150provide API more familiar and easier to use:
151
152```elixir
153defmodule Commoner.ExampleSuite do
154 use CommonTest.Suite
155
156 test "foo bar" do
157 assert (1 + 1) in [2]
158 end
159
160 test "bar baz" do
161 CommonTest.log("Hello")
162 end
163end
164```
165
166I am very interested about your feelings about Common Test usage in Elixir and
167about proposed API for Commoner. You can ping me:
168
169- On [Elixir forum thread about Commoner][forum]
170- On Twitter via `@hauleth`
171
172### Special thanks
173
174- José Valim - for reading it through and reviewing before publishing
175
176[ct]: http://www.erlang.org/doc/man/basics_chapter.html
177[ctex]: https://github.com/Comcast/ctex
178[commoner]: https://github.com/hauleth/commoner
179[forum]: https://elixirforum.com/t/commoner-elixir-wrapper-for-common-test-library/23966?u=hauleth