this repo has no description
hauleth.dev
1+++
2title = "Common Test for Elixir developers"
3date = 2019-07-15
4
5[taxonomies]
6tags = [
7 "beam",
8 "testing"
9]
10
11[[extra.thanks]]
12name = "José Valim"
13why = "for reading it through and reviewing before publishing"
14+++
15
16In my new job I have opportunity to work a little bit more with Erlang code and
17Erlang related tools than earlier. That forced me a little to learn and use
18[Common Test][ct] library which is **the** testing library in Erlang world.
19When after that I needed to work back it hit me hard how much stuff I am
20missing in Elixir. In this article I will try to present Common Test library
21from the viewpoint of long standing Elixir guy and present how it compares to
22the ExUnit.
23
24## Unit testing vs integration testing
25
26The main difference between these two is their intended usage. In it's core
27ExUnit is intended as a library for unit testing (no surprise there, as it is
28literally in it's name), on the other hand Common Test is more about integration
29testing and looking on the system as a whole. So the discussion there would not
30be fair, as it isn't apples to apples, but for second let's check what makes
31each of them great for their intended uses:
32
33ExUnit for unit testing:
34
35- Simple way to setup and teardown tests via `setup` and `setup_all` macros
36- Built in testing of documentation examples via `doctest`
37- Immutability of test environment (at least that is what you should try to
38 achieve)
39- Very simple way of running different test modules in parallel (tests within
40 single module are always sync)
41- Each test run in separate process, which prevents sharing data between
42 processes via process dictionary
43- Grouping tests into `describe` blocks
44- In overall simplicity of the library as a whole
45
46Common Test for integration testing:
47
48- Persistent test results - logs, results, "generated garbage" is stored between
49 runs in the Common Test, which allows to track what and when happened, and if
50 needed trace the bug in the logs/leftovers
51- Generating "private directory" per suite
52- Grouping tests while allowing single test to be reused in different groups
53- Extensive control about order and parallelization of tests within groups
54- Built in very extensive and detailed reports (HTML and JUnit)
55- Distributed testing
56- Define dependencies between tests
57- Set of helpers for working with different network protocols (SSH, Telnet,
58 SNMP, FTP, Erlang RPC, netconf)
59- Built in support for data fixtures
60- Built in support for complex configuration of test runs via [test
61 specifications](http://erlang.org/doc/apps/common_test/run_test_chapter.html#test-specifications)
62- Built in logging (we will cover that later)
63
64So as you can see, the Common Test is much broader and complex, but at the same
65time it provides a lot of utilities that are really helpful while writing
66integration tests, that can span multiple applications.
67
68## Writing Common Test suite
69
70While CT provides enormous power in hand of the experienced tester/developer
71it's API isn't the top of the class (which happen a lot in Erlang tools, whoever
72wanted to integrate them with anything then they will know what I mean). But in
73fact writing simple test suite in CT is fairly easy:
74
75```erlang
76-module(example_SUITE). % The naming convention (with uppercase _SUITE) Erlang
77 % convention which allow ct to find test suites.
78 % Something like ExUnit _test.exs naming convention
79
80-export([all/0]).
81
82-export([test_function_name/1]).
83
84all() ->
85 [test_function_name].
86
87test_function_name(_Config) ->
88 2 = 1 + 1.
89```
90
91It is pretty easy and understandable format, but with due CT power it [quickly
92can grow to be "less" readable](https://github.com/erlang/otp/blob/81d332cc693d2be8a5af16bfbcae0bfde6702479/lib/ssh/test/ssh_algorithms_SUITE.erl#L36).
93Taming that power can be problematic and can cause few headaches for newcomers.
94
95## Reporting
96
97Another part, and the one that I miss the most in ExUnit, is reporting in Common
98Test. This alone is in my humble opinion **THE** best part of the library. To
99present you how this work assume that we have above test suite which we will run
100with `ct_run -dir .`. This will compile and run our tests (obviously), but in
101addition to that it will generate hell lot of cruft files, some JQuery (yes,
102this is still used in 2019), some CSS files, some HTML and other.
103
104Just check [this out](/common-test-example/simple/index.html). This is example report
105generated by the Common Test. As you can see it contains a lot of information in
106quite readable format. Not only it contains information about current run, but
107all previous runs as well, which is really handy when tracking regressions.
108
109But can we store even more information there? Yes, as CT includes simple
110logging facility it is completely possible to log your own information during
111tests, for example, lets modify our test to log some information:
112
113```erlang
114test_function_name(_Config) ->
115 ct:log("Example message"),
116 2 = 1 + 1.
117```
118
119Now when we run tests again, then we will see more information (even coloured)
120in [our test log](/common-test-example/log/ct_run.ct@NiunioBook.2019-07-16_11.03.21/common-test-example.log.logs/run.2019-07-16_11.03.22/example_suite.test_function_name.html):
121
122
123
124Additionally there is support Surefire XML output (commonly known as JUnit XML)
125via [hook that is distributed with Common Test](http://erlang.org/doc/apps/common_test/ct_hooks_chapter.html#built-in-cths).
126This is very useful, as many CI services (for sure Jenkins, Circle CI, and
127GitLab CI) support such files to provide better CI results than raw stdout.
128
129## Comparison to ExUnit
130
131ExUnit is great tool, which is very flexible while being relatively simple.
132Unfortunately that simplicity makes it very lacking when comparing with "more
133established" solutions like Common Test which shines when we "grow out" of the
134ExUnit tests. It provides a lot great tools that really help with finding bugs
135(for example when I was joining the project and I was trying to run tests
136locally I have some issues due to misconfiguration, it was much easier to send
137tarball of the test results instead of sending wall of text captured from
138stdout).
139
140Don't get me wrong, ExUnit is great and powerful, and for sure you should use
141it. It is great evolution over EUnit, but again, it would be worth of having
142built in [Surefire XML output](http://www.erlang.org/doc/man/eunit_surefire.html)
143due to it's popularity and support in many CI services.
144
145## Future
146
147But is everything lost and we need to fall back to the Erlang code (which can be
148problematic for many people) to use Common Test? Not exactly, Comcast (yes, that
149Comcast) [created simple wrapper][ctex] for Elixir (unfortunately seems pretty
150dead to me) and I have started [Commoner][commoner] library that is meant to
151provide API more familiar and easier to use:
152
153```elixir
154defmodule Commoner.ExampleSuite do
155 use CommonTest.Suite
156
157 test "foo bar" do
158 assert (1 + 1) in [2]
159 end
160
161 test "bar baz" do
162 CommonTest.log("Hello")
163 end
164end
165```
166
167I am very interested about your feelings about Common Test usage in Elixir and
168about proposed API for Commoner. You can ping me:
169
170- On [Elixir forum thread about Commoner][forum]
171- On Twitter via `@hauleth`
172
173[ct]: http://erlang.org/doc/apps/common_test/basics_chapter.html
174[ctex]: https://github.com/Comcast/ctex
175[commoner]: https://github.com/hauleth/commoner
176[forum]: https://elixirforum.com/t/commoner-elixir-wrapper-for-common-test-library/23966?u=hauleth