···
2
+
title = "Who Watches Watchmen? - Part 2"
6
+
Continuation of travel into making systemd to work for us, not against us. This
7
+
time we will talk about socket activation and how to make our application run
8
+
only when we need it to run.
21
+
why = "Helping me with my poor English"
24
+
This is continuation of [Part I][part-i] where I described the basics of the
25
+
supervising BEAM applications with systemd and how to create basic, secure
26
+
service for your Elixir application with it. In this article I will assume that
27
+
you have read [the previous one][part-i].
29
+
______________________________________________________________________
31
+
We already have our super simple service description. Just to refresh your
32
+
memory, it is the `hello.service` file once again:
36
+
Description=Hello World service
37
+
Requires=network.target
42
+
ExecStart=/opt/hello/bin/hello start
45
+
# We need to add capability to be able to bind on port 80
46
+
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
51
+
Environment=ERL_CRASH_DUMP_SECONDS=0
54
+
However there is one small problem. It allows our service to listen on **any**
55
+
restricted port, not just `80` that we want to listen on. This can be
56
+
troublesome as an attacker that gains RCE on our server can then capture any
57
+
traffic on any port that we do not want to open (for example exposing port 22
58
+
using the [`ssh`] module).
60
+
It would be nice if we could somehow inject sockets for only the ports we want
61
+
to listen to into our application.
65
+
Thanks to the [`systemd.socket`][systemd.socket] feature we can achieve that
66
+
with a little work on our side.
68
+
First we need to create new unit named `hello.socket` next to our
73
+
Description=Listening socket
74
+
Requires=sockets.target
83
+
It will create a socket connected to TCP 80 (because we used `ListenStream=`,
84
+
and TCP is the stream protocol). By default it will bind that socket to a
85
+
service named the same as our socket, so now we need to edit our `hello.service`
90
+
Description=Hello World service
91
+
Requires=network.target
96
+
ExecStart=/opt/hello/bin/hello start
99
+
# See, we no longer need to insecurely allow binding to any port
100
+
# CapabilityBoundingSet=CAP_NET_BIND_SERVICE
104
+
PrivateDevices=true
105
+
Environment=ERL_CRASH_DUMP_SECONDS=0
108
+
And we need to modify our `Hello.Application.cowboy_opts/0` to handle the socket
109
+
which is passed to us a file descriptor:
112
+
# hello/application.ex
113
+
defmodule Hello.Application do
116
+
def start(_type, _opts) do
117
+
fds = :systemd.listen_fds()
120
+
{Plug.Cowboy, [scheme: :http, plug: Hello.Router] ++ cowboy_opts(fds)},
121
+
{Plug.Cowboy.Drainer, refs: :all}
124
+
Supervisor.start_link(children, strategy: :one_for_one)
127
+
# If there are no sockets passed to the application, then start listening on
128
+
# the port specified by the `PORT` environment variable
129
+
defp cowboy_opts([]) do
130
+
[port: String.to_integer(System.get_env("PORT", "5000"))]
133
+
# If there are any socket passed, then use first one
134
+
defp cowboy_opts([socket | _]) do
137
+
# Sockets can be named, which will be passed as the second element in
140
+
# Or unnamed, and then it will be just the file descriptor
153
+
1. Systemd sockets are IPv6 enabled (we explicitly said that we want to listen
154
+
on both). That means, that we need to mark our connection as an INET6
155
+
connection. This will not affect IPv4 (INET) connections.
156
+
1. We are required to pass `:port` key, but its value will be ignored, so we
158
+
1. We pass the file descriptor that will be then passed to the Cowboy listener.
160
+
Now when we will start our service:
163
+
# systemctl start hello.service
166
+
It will be available at `https://localhost/` while still running as an
171
+
The question may arise - how to allow our service to listen on more than one
172
+
port, for example you want to have your website available as HTTPS alongside
173
+
"regular" HTTP. This means that our application needs to listen on two
179
+
Now we need to slightly modify a little our socket service and add another one.
180
+
First rename our `hello.socket` to `hello-http.socket` and add a line
181
+
`Service=hello.service` and `FileDescriptorName=http` to `[Socket]` section, so
186
+
Description=HTTP Socket
187
+
Requires=sockets.target
190
+
# We declare the name of the file descriptor here to simplify extraction in
191
+
# the application afterwards. By default it will be the socket name (so
192
+
# `hello-http` in our case), but `http` is much cleaner.
193
+
FileDescriptorName=http
195
+
Service=hello.service
201
+
Next we create a similar file, but for HTTPS named `hello-https.socket`
205
+
Description=HTTPS Socket
206
+
Requires=sockets.target
209
+
FileDescriptorName=https
211
+
Service=hello.service
217
+
And we add the dependency on both of our sockets to the `hello.service`:
221
+
Description=Hello World service
222
+
After=hello-http.socket hello-https.socket
223
+
BindTo=hello-http.socket hello-https.socket
226
+
ExecStart=/opt/hello/bin/hello start
230
+
PrivateDevices=true
231
+
Environment=ERL_CRASH_DUMB_SECONDS=0
234
+
Now we need to somehow differentiate between our sockets in the
235
+
`Hello.Application`, so we will be able to pass the proper FD to each of the
236
+
listeners. The `:systemd.listen_fds/0` will return a list of file descriptors,
237
+
and if they are named, the format will be a 2-tuple where the first element is
238
+
the file descriptor and the second is the name as a string:
241
+
# hello/application.ex
242
+
defmodule Hello.Application do
245
+
def start(_type, _opts) do
246
+
fds = :systemd.listen_fds()
248
+
router = Hello.Router
254
+
] ++ cowboy_opts(fds, "http")},
258
+
keyfile: "path/to/keyfile.pem",
259
+
certfile: "path/to/certfile.pem",
260
+
dhfile: "path/to/dhfile.pem"
261
+
] ++ cowboy_opts(fds, "https")},
262
+
{Plug.Cowboy.Drainer, refs: :all}
265
+
Supervisor.start_link(children, strategy: :one_for_one)
268
+
defp cowboy_opts(fds, protocol) do
269
+
case List.keyfind(fds, protocol, 1) do
270
+
# If there is socket passed for given protocol, then use that one
278
+
# If there are no sockets passed to the application that match
279
+
# the protocol, then start listening on the port specified by
280
+
# `PORT_{protocol}` environment variable
283
+
port: String.to_integer(System.get_env("PORT_#{protocol}", "5000"))
289
+
Now our application will listen on both - HTTP and HTTPS, despite running as
292
+
## Socket activation
294
+
Now, that we can inject sockets to our application with ease we can achieve even
295
+
more fascinating feature - socket activation.
297
+
Some of you may used `inetd` in the past, that allows you to dynamically start
298
+
processes on network requests. It is quite an interesting tool that detects
299
+
traffic on certain ports, then spawns a new process to handle it, passing data
300
+
to and from that process via `STDIN` and `STDOUT`. There was a quirk though, it
301
+
required the spawned process to shutdown after it handled the request and it was
302
+
starting a new instance for each request. That works poorly with VMs like BEAM
303
+
that have substantial startup time and are expected to be long-running systems.
304
+
BEAM is capable of handling network requests on it's own.
306
+
Fortunately for us, the way that we have implemented our systemd service is all
307
+
that we need to have our application dynamically activated. To observe that we
308
+
just need to shutdown everything:
311
+
# systemctl stop hello-http.socket hello-https.socket hello.service
314
+
And now relaunch **only the sockets**:
317
+
# systemctl start hello-http.socket hello-https.socket
320
+
We can check, that our service is not running:
323
+
$ systemctl status hello.service
324
+
● hello.service - Hello World service
325
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
326
+
Active: inactive (dead)
327
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
330
+
We can see the `TriggeredBy` section that tells us, that this service will be
331
+
started by one of the sockets listed there. Let see what will happen when we
332
+
will try to request anything from our application:
335
+
$ curl http://localhost/
339
+
You can see that we got a response from our application. This mean that our
340
+
application must have started, and indeed when we check:
343
+
$ systemctl status hello.service
344
+
● hello.service - Hello
345
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
346
+
Active: active (running) since Thu 2022-02-03 13:20:27 CET; 4s ago
347
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
348
+
Main PID: 1106 (beam.smp)
349
+
Tasks: 19 (limit: 1136)
351
+
CGroup: /system.slice/hello.service
352
+
├─1106 /opt/hello/erts-12.2/bin/beam.smp -- -root /opt/hello -progname erl -- -home /run/hello -- -noshell -s elixir start_cli -mode embedded -setcookie CR63SVI6L5JAMJSDL3H4XPNMOPHEWSV2FPHCHCAN65CY6ASHMXBA==== -sname hello -c>
353
+
└─1138 erl_child_setup 1024
356
+
It seems to be running, and if we stop it, then we will get information that it
357
+
still can be activated by our sockets:
360
+
# systemctl stop hello.service
361
+
Warning: Stopping hello.service, but it can still be activated by:
362
+
hello-http.socket hello-https.socket
365
+
That means, that systemd is still listening on the sockets that we defined, even
366
+
when our application is down, and will start our application again as soon as
367
+
there are any incoming requests.
369
+
Let test that out again:
372
+
$ curl http://localhost/
374
+
$ systemctl status hello.service
375
+
● hello.service - Hello
376
+
Loaded: loaded (/usr/local/lib/systemd/system/hello.service; static; vendor preset: enabled)
377
+
Active: active (running) since Thu 2022-02-03 13:22:27 CET; 4s ago
378
+
TriggeredBy: ● hello-http.socket ● hello-https.socket
379
+
Main PID: 3452 (beam.smp)
380
+
Tasks: 19 (limit: 1136)
382
+
CGroup: /system.slice/hello.service
383
+
├─3452 /opt/hello/erts-12.2/bin/beam.smp -- -root /opt/hello -progname erl -- -home /run/hello -- -noshell -s elixir start_cli -mode embedded -setcookie CR63SVI6L5JAMJSDL3H4XPNMOPHEWSV2FPHCHCAN65CY6ASHMXBA==== -sname hello -c>
384
+
└─3453 erl_child_setup 1024
387
+
Our application got launched again, automatically, just by the fact that
388
+
there was incoming TCP connection.
390
+
Does it work for HTTPS connection as well?
393
+
# systemctl stop hello.service
394
+
$ curl -k https://localhost/
398
+
It seems so. Independently of which port we try to reach our application on, it
399
+
will be automatically launched for us and the connection will be properly
400
+
handled. Do note that systemd will not shut down our process after serving the
401
+
request. It will continue to run from that point forward.
405
+
I know that it took quite while since the last post (ca. 1.5 years), but I hope
406
+
that I will be able to write the final part much sooner than this.
408
+
- [Part 1 - Basics, security, and FD passing][part-i]
409
+
- [Part 2 - Socket activation (this one)](./#top)
412
+
[part-i]: @/post/who-watches-watchmen-i.md
413
+
[systemd.socket]: https://www.freedesktop.org/software/systemd/man/systemd.socket.html
414
+
[`ssh`]: https://erlang.org/doc/man/ssh.html
hauleth.dev