pyrox.dev
1<chapter xmlns="http://docbook.org/ns/docbook"
2 xmlns:xlink="http://www.w3.org/1999/xlink"
3 xmlns:xi="http://www.w3.org/2001/XInclude"
4 version="5.0"
5 xml:id="module-services-foundationdb">
6
7<title>FoundationDB</title>
8
9<para><emphasis>Source:</emphasis> <filename>modules/services/databases/foundationdb.nix</filename></para>
10
11<para><emphasis>Upstream documentation:</emphasis> <link xlink:href="https://apple.github.io/foundationdb/"/></para>
12
13<para><emphasis>Maintainer:</emphasis> Austin Seipp</para>
14
15<para><emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x</para>
16
17<para>FoundationDB (or "FDB") is an open source, distributed, transactional
18key-value store.</para>
19
20<section xml:id="module-services-foundationdb-configuring"><title>Configuring and basic setup</title>
21
22<para>To enable FoundationDB, add the following to your
23<filename>configuration.nix</filename>:
24
25<programlisting>
26services.foundationdb.enable = true;
27services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
28</programlisting>
29</para>
30
31<para>The <option>services.foundationdb.package</option> option is required,
32and must always be specified. Due to the fact FoundationDB network protocols and
33on-disk storage formats may change between (major) versions, and upgrades must
34be explicitly handled by the user, you must always manually specify this
35yourself so that the NixOS module will use the proper version. Note that minor,
36bugfix releases are always compatible.</para>
37
38<para>After running <command>nixos-rebuild</command>, you can verify whether
39FoundationDB is running by executing <command>fdbcli</command> (which is added
40to <option>environment.systemPackages</option>):
41
42<programlisting>
43$ sudo -u foundationdb fdbcli
44Using cluster file `/etc/foundationdb/fdb.cluster'.
45
46The database is available.
47
48Welcome to the fdbcli. For help, type `help'.
49fdb> status
50
51Using cluster file `/etc/foundationdb/fdb.cluster'.
52
53Configuration:
54 Redundancy mode - single
55 Storage engine - memory
56 Coordinators - 1
57
58Cluster:
59 FoundationDB processes - 1
60 Machines - 1
61 Memory availability - 5.4 GB per process on machine with least available
62 Fault Tolerance - 0 machines
63 Server time - 04/20/18 15:21:14
64
65...
66
67fdb>
68</programlisting>
69</para>
70
71<para>You can also write programs using the available client libraries.
72For example, the following Python program can be run in order to grab the
73cluster status, as a quick example. (This example uses
74<command>nix-shell</command> shebang support to automatically supply the
75necessary Python modules).
76
77<programlisting>
78a@link> cat fdb-status.py
79#! /usr/bin/env nix-shell
80#! nix-shell -i python -p python pythonPackages.foundationdb52
81
82import fdb
83import json
84
85def main():
86 fdb.api_version(520)
87 db = fdb.open()
88
89 @fdb.transactional
90 def get_status(tr):
91 return str(tr['\xff\xff/status/json'])
92
93 obj = json.loads(get_status(db))
94 print('FoundationDB available: %s' % obj['client']['database_status']['available'])
95
96if __name__ == "__main__":
97 main()
98a@link> chmod +x fdb-status.py
99a@link> ./fdb-status.py
100FoundationDB available: True
101a@link>
102</programlisting>
103</para>
104
105<para>FoundationDB is run under the <command>foundationdb</command> user and
106group by default, but this may be changed in the NixOS configuration. The
107systemd unit <command>foundationdb.service</command> controls the
108<command>fdbmonitor</command> process.</para>
109
110<para>By default, the NixOS module for FoundationDB creates a single
111SSD-storage based database for development and basic usage. This storage engine
112is designed for SSDs and will perform poorly on HDDs; however it can handle far
113more data than the alternative "memory" engine and is a better default choice
114for most deployments. (Note that you can change the storage backend on-the-fly
115for a given FoundationDB cluster using <command>fdbcli</command>.)</para>
116
117<para>Furthermore, only 1 server process and 1 backup agent are started in the
118default configuration. See below for more on scaling to increase this.</para>
119
120<para>FoundationDB stores all data for all server processes under
121<filename>/var/lib/foundationdb</filename>. You can override this using
122<option>services.foundationdb.dataDir</option>, e.g.
123
124<programlisting>
125services.foundationdb.dataDir = "/data/fdb";
126</programlisting>
127
128</para>
129
130<para>Similarly, logs are stored under
131<filename>/var/log/foundationdb</filename> by default, and there is a
132corresponding <option>services.foundationdb.logDir</option> as well.</para>
133
134</section>
135
136<section xml:id="module-services-foundationdb-scaling"><title>Scaling processes and backup agents</title>
137
138<para>Scaling the number of server processes is quite easy; simply specify
139<option>services.foundationdb.serverProcesses</option> to be the number of
140FoundationDB worker processes that should be started on the machine.</para>
141
142<para>FoundationDB worker processes typically require 4GB of RAM per-process at
143minimum for good performance, so this option is set to 1 by default since the
144maximum amount of RAM is unknown. You're advised to abide by this restriction,
145so pick a number of processes so that each has 4GB or more.</para>
146
147<para>A similar option exists in order to scale backup agent processes,
148<option>services.foundationdb.backupProcesses</option>. Backup agents are not
149as performance/RAM sensitive, so feel free to experiment with the number of
150available backup processes.</para>
151
152</section>
153
154<section xml:id="module-services-foundationdb-clustering"><title>Clustering</title>
155
156<para>FoundationDB on NixOS works similarly to other Linux systems, so this
157section will be brief. Please refer to the full FoundationDB documentation for
158more on clustering.</para>
159
160<para>FoundationDB organizes clusters using a set of
161<emphasis>coordinators</emphasis>, which are just specially-designated worker
162processes. By default, every installation of FoundationDB on NixOS will start
163as its own individual cluster, with a single coordinator: the first worker
164process on <command>localhost</command>.</para>
165
166<para>Coordinators are specified globally using the
167<command>/etc/foundationdb/fdb.cluster</command> file, which all servers and
168client applications will use to find and join coordinators. Note that this file
169<emphasis>can not</emphasis> be managed by NixOS so easily: FoundationDB is
170designed so that it will rewrite the file at runtime for all clients and nodes
171when cluster coordinators change, with clients transparently handling this
172without intervention. It is fundamentally a mutable file, and you should not
173try to manage it in any way in NixOS.</para>
174
175<para>When dealing with a cluster, there are two main things you want to
176do:</para>
177
178<itemizedlist>
179 <listitem><para>Add a node to the cluster for storage/compute.</para></listitem>
180 <listitem><para>Promote an ordinary worker to a coordinator.</para></listitem>
181</itemizedlist>
182
183<para>A node must already be a member of the cluster in order to properly be
184promoted to a coordinator, so you must always add it first if you wish to
185promote it.</para>
186
187<para>To add a machine to a FoundationDB cluster:</para>
188
189<itemizedlist>
190 <listitem><para>Choose one of the servers to start as the initial coordinator.
191 </para></listitem>
192 <listitem><para>Copy the <command>/etc/foundationdb/fdb.cluster</command> file
193 from this server to all the other servers. Restart FoundationDB on all of
194 these other servers, so they join the cluster.</para></listitem>
195 <listitem><para>All of these servers are now connected and working together
196 in the cluster, under the chosen coordinator.</para></listitem>
197</itemizedlist>
198
199<para>At this point, you can add as many nodes as you want by just repeating
200the above steps. By default there will still be a single coordinator: you can
201use <command>fdbcli</command> to change this and add new coordinators.</para>
202
203<para>As a convenience, FoundationDB can automatically assign coordinators
204based on the redundancy mode you wish to achieve for the cluster. Once all the
205nodes have been joined, simply set the replication policy, and then issue the
206<command>coordinators auto</command> command</para>
207
208<para>For example, assuming we have 3 nodes available, we can enable double
209redundancy mode, then auto-select coordinators. For double redundancy, 3
210coordinators is ideal: therefore FoundationDB will make
211<emphasis>every</emphasis> node a coordinator automatically:</para>
212
213<programlisting>
214fdbcli> configure double ssd
215fdbcli> coordinators auto
216</programlisting>
217
218<para>This will transparently update all the servers within seconds, and
219appropriately rewrite the <command>fdb.cluster</command> file, as well as
220informing all client processes to do the same.</para>
221
222</section>
223
224<section xml:id="module-services-foundationdb-connectivity"><title>Client connectivity</title>
225
226<para>By default, all clients must use the current
227<command>fdb.cluster</command> file to access a given FoundationDB cluster.
228This file is located by default in
229<command>/etc/foundationdb/fdb.cluster</command> on all machines with the
230FoundationDB service enabled, so you may copy the active one from your cluster
231to a new node in order to connect, if it is not part of the cluster.</para>
232
233</section>
234
235<section xml:id="module-services-foundationdb-authorization"><title>Client authorization and TLS</title>
236
237<para>By default, any user who can connect to a FoundationDB process with the
238correct cluster configuration can access anything. FoundationDB uses a
239pluggable design to transport security, and out of the box it supports a
240LibreSSL-based plugin for TLS support. This plugin not only does in-flight
241encryption, but also performs client authorization based on the given
242endpoint's certificate chain. For example, a FoundationDB server may be
243configured to only accept client connections over TLS, where the client TLS
244certificate is from organization <emphasis>Acme Co</emphasis> in the
245<emphasis>Research and Development</emphasis> unit.</para>
246
247<para>Configuring TLS with FoundationDB is done using the
248<option>services.foundationdb.tls</option> options in order to control the peer
249verification string, as well as the certificate and its private key.</para>
250
251<para>Note that the certificate and its private key must be accessible to the
252FoundationDB user account that the server runs under. These files are also NOT
253managed by NixOS, as putting them into the store may reveal private
254information.</para>
255
256<para>After you have a key and certificate file in place, it is not enough to
257simply set the NixOS module options -- you must also configure the
258<command>fdb.cluster</command> file to specify that a given set of coordinators
259use TLS. This is as simple as adding the suffix <command>:tls</command> to your
260cluster coordinator configuration, after the port number. For example, assuming
261you have a coordinator on localhost with the default configuration, simply
262specifying:</para>
263
264<programlisting>
265XXXXXX:XXXXXX@127.0.0.1:4500:tls
266</programlisting>
267
268<para>will configure all clients and server processes to use TLS from now
269on.</para>
270
271</section>
272
273<section xml:id="module-services-foundationdb-disaster-recovery"><title>Backups and Disaster Recovery</title>
274
275<para>The usual rules for doing FoundationDB backups apply on NixOS as written
276in the FoundationDB manual. However, one important difference is the security
277profile for NixOS: by default, the <command>foundationdb</command> systemd unit
278uses <emphasis>Linux namespaces</emphasis> to restrict write access to the
279system, except for the log directory, data directory, and the
280<command>/etc/foundationdb/</command> directory. This is enforced by default
281and cannot be disabled.</para>
282
283<para>However, a side effect of this is that the <command>fdbbackup</command>
284command doesn't work properly for local filesystem backups: FoundationDB uses a
285server process alongside the database processes to perform backups and copy the
286backups to the filesystem. As a result, this process is put under the
287restricted namespaces above: the backup process can only write to a limited
288number of paths.</para>
289
290<para>In order to allow flexible backup locations on local disks, the
291FoundationDB NixOS module supports a
292<option>services.foundationdb.extraReadWritePaths</option> option. This option
293takes a list of paths, and adds them to the systemd unit, allowing the
294processes inside the service to write (and read) the specified
295directories.</para>
296
297<para>For example, to create backups in <command>/opt/fdb-backups</command>,
298first set up the paths in the module options:</para>
299
300<programlisting>
301services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
302</programlisting>
303
304<para>Restart the FoundationDB service, and it will now be able to write to
305this directory (even if it does not yet exist.) Note: this path
306<emphasis>must</emphasis> exist before restarting the unit. Otherwise, systemd
307will not include it in the private FoundationDB namespace (and it will not add
308it dynamically at runtime).</para>
309
310<para>You can now perform a backup:</para>
311
312<programlisting>
313$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
314$ sudo -u foundationdb fdbbackup status -t default
315</programlisting>
316
317</section>
318
319<section xml:id="module-services-foundationdb-limitations"><title>Known limitations</title>
320
321<para>The FoundationDB setup for NixOS should currently be considered beta.
322FoundationDB is not new software, but the NixOS compilation and integration has
323only undergone fairly basic testing of all the available functionality.</para>
324
325<itemizedlist>
326 <listitem><para>There is no way to specify individual parameters for
327 individual <command>fdbserver</command> processes. Currently, all server
328 processes inherit all the global <command>fdbmonitor</command> settings.
329 </para></listitem>
330 <listitem><para>Ruby bindings are not currently installed.</para></listitem>
331 <listitem><para>Go bindings are not currently installed.</para></listitem>
332</itemizedlist>
333
334</section>
335
336<section xml:id="module-services-foundationdb-options"><title>Options</title>
337
338<para>NixOS's FoundationDB module allows you to configure all of the most
339relevant configuration options for <command>fdbmonitor</command>, matching it
340quite closely. A complete list of options for the FoundationDB module may be
341found <link linkend="opt-services.foundationdb.enable">here</link>. You should
342also read the FoundationDB documentation as well.</para>
343
344</section>
345
346<section xml:id="module-services-foundationdb-full-docs"><title>Full documentation</title>
347
348<para>FoundationDB is a complex piece of software, and requires careful
349administration to properly use. Full documentation for administration can be
350found here: <link xlink:href="https://apple.github.io/foundationdb/"/>.</para>
351
352</section>
353
354</chapter>