pyrox.dev
1# FoundationDB {#module-services-foundationdb}
2
3*Source:* {file}`modules/services/databases/foundationdb.nix`
4
5*Upstream documentation:* <https://apple.github.io/foundationdb/>
6
7*Maintainer:* Austin Seipp
8
9*Available version(s):* 5.1.x, 5.2.x, 6.0.x
10
11FoundationDB (or "FDB") is an open source, distributed, transactional
12key-value store.
13
14## Configuring and basic setup {#module-services-foundationdb-configuring}
15
16To enable FoundationDB, add the following to your
17{file}`configuration.nix`:
18```
19services.foundationdb.enable = true;
20services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
21```
22
23The {option}`services.foundationdb.package` option is required, and
24must always be specified. Due to the fact FoundationDB network protocols and
25on-disk storage formats may change between (major) versions, and upgrades
26must be explicitly handled by the user, you must always manually specify
27this yourself so that the NixOS module will use the proper version. Note
28that minor, bugfix releases are always compatible.
29
30After running {command}`nixos-rebuild`, you can verify whether
31FoundationDB is running by executing {command}`fdbcli` (which is
32added to {option}`environment.systemPackages`):
33```ShellSession
34$ sudo -u foundationdb fdbcli
35Using cluster file `/etc/foundationdb/fdb.cluster'.
36
37The database is available.
38
39Welcome to the fdbcli. For help, type `help'.
40fdb> status
41
42Using cluster file `/etc/foundationdb/fdb.cluster'.
43
44Configuration:
45 Redundancy mode - single
46 Storage engine - memory
47 Coordinators - 1
48
49Cluster:
50 FoundationDB processes - 1
51 Machines - 1
52 Memory availability - 5.4 GB per process on machine with least available
53 Fault Tolerance - 0 machines
54 Server time - 04/20/18 15:21:14
55
56...
57
58fdb>
59```
60
61You can also write programs using the available client libraries. For
62example, the following Python program can be run in order to grab the
63cluster status, as a quick example. (This example uses
64{command}`nix-shell` shebang support to automatically supply the
65necessary Python modules).
66```ShellSession
67a@link> cat fdb-status.py
68#! /usr/bin/env nix-shell
69#! nix-shell -i python -p python pythonPackages.foundationdb52
70
71import fdb
72import json
73
74def main():
75 fdb.api_version(520)
76 db = fdb.open()
77
78 @fdb.transactional
79 def get_status(tr):
80 return str(tr['\xff\xff/status/json'])
81
82 obj = json.loads(get_status(db))
83 print('FoundationDB available: %s' % obj['client']['database_status']['available'])
84
85if __name__ == "__main__":
86 main()
87a@link> chmod +x fdb-status.py
88a@link> ./fdb-status.py
89FoundationDB available: True
90a@link>
91```
92
93FoundationDB is run under the {command}`foundationdb` user and group
94by default, but this may be changed in the NixOS configuration. The systemd
95unit {command}`foundationdb.service` controls the
96{command}`fdbmonitor` process.
97
98By default, the NixOS module for FoundationDB creates a single SSD-storage
99based database for development and basic usage. This storage engine is
100designed for SSDs and will perform poorly on HDDs; however it can handle far
101more data than the alternative "memory" engine and is a better default
102choice for most deployments. (Note that you can change the storage backend
103on-the-fly for a given FoundationDB cluster using
104{command}`fdbcli`.)
105
106Furthermore, only 1 server process and 1 backup agent are started in the
107default configuration. See below for more on scaling to increase this.
108
109FoundationDB stores all data for all server processes under
110{file}`/var/lib/foundationdb`. You can override this using
111{option}`services.foundationdb.dataDir`, e.g.
112```
113services.foundationdb.dataDir = "/data/fdb";
114```
115
116Similarly, logs are stored under {file}`/var/log/foundationdb`
117by default, and there is a corresponding
118{option}`services.foundationdb.logDir` as well.
119
120## Scaling processes and backup agents {#module-services-foundationdb-scaling}
121
122Scaling the number of server processes is quite easy; simply specify
123{option}`services.foundationdb.serverProcesses` to be the number of
124FoundationDB worker processes that should be started on the machine.
125
126FoundationDB worker processes typically require 4GB of RAM per-process at
127minimum for good performance, so this option is set to 1 by default since
128the maximum amount of RAM is unknown. You're advised to abide by this
129restriction, so pick a number of processes so that each has 4GB or more.
130
131A similar option exists in order to scale backup agent processes,
132{option}`services.foundationdb.backupProcesses`. Backup agents are
133not as performance/RAM sensitive, so feel free to experiment with the number
134of available backup processes.
135
136## Clustering {#module-services-foundationdb-clustering}
137
138FoundationDB on NixOS works similarly to other Linux systems, so this
139section will be brief. Please refer to the full FoundationDB documentation
140for more on clustering.
141
142FoundationDB organizes clusters using a set of
143*coordinators*, which are just specially-designated
144worker processes. By default, every installation of FoundationDB on NixOS
145will start as its own individual cluster, with a single coordinator: the
146first worker process on {command}`localhost`.
147
148Coordinators are specified globally using the
149{command}`/etc/foundationdb/fdb.cluster` file, which all servers and
150client applications will use to find and join coordinators. Note that this
151file *can not* be managed by NixOS so easily:
152FoundationDB is designed so that it will rewrite the file at runtime for all
153clients and nodes when cluster coordinators change, with clients
154transparently handling this without intervention. It is fundamentally a
155mutable file, and you should not try to manage it in any way in NixOS.
156
157When dealing with a cluster, there are two main things you want to do:
158
159 - Add a node to the cluster for storage/compute.
160 - Promote an ordinary worker to a coordinator.
161
162A node must already be a member of the cluster in order to properly be
163promoted to a coordinator, so you must always add it first if you wish to
164promote it.
165
166To add a machine to a FoundationDB cluster:
167
168 - Choose one of the servers to start as the initial coordinator.
169 - Copy the {command}`/etc/foundationdb/fdb.cluster` file from this
170 server to all the other servers. Restart FoundationDB on all of these
171 other servers, so they join the cluster.
172 - All of these servers are now connected and working together in the
173 cluster, under the chosen coordinator.
174
175At this point, you can add as many nodes as you want by just repeating the
176above steps. By default there will still be a single coordinator: you can
177use {command}`fdbcli` to change this and add new coordinators.
178
179As a convenience, FoundationDB can automatically assign coordinators based
180on the redundancy mode you wish to achieve for the cluster. Once all the
181nodes have been joined, simply set the replication policy, and then issue
182the {command}`coordinators auto` command
183
184For example, assuming we have 3 nodes available, we can enable double
185redundancy mode, then auto-select coordinators. For double redundancy, 3
186coordinators is ideal: therefore FoundationDB will make
187*every* node a coordinator automatically:
188
189```ShellSession
190fdbcli> configure double ssd
191fdbcli> coordinators auto
192```
193
194This will transparently update all the servers within seconds, and
195appropriately rewrite the {command}`fdb.cluster` file, as well as
196informing all client processes to do the same.
197
198## Client connectivity {#module-services-foundationdb-connectivity}
199
200By default, all clients must use the current {command}`fdb.cluster`
201file to access a given FoundationDB cluster. This file is located by default
202in {command}`/etc/foundationdb/fdb.cluster` on all machines with the
203FoundationDB service enabled, so you may copy the active one from your
204cluster to a new node in order to connect, if it is not part of the cluster.
205
206## Client authorization and TLS {#module-services-foundationdb-authorization}
207
208By default, any user who can connect to a FoundationDB process with the
209correct cluster configuration can access anything. FoundationDB uses a
210pluggable design to transport security, and out of the box it supports a
211LibreSSL-based plugin for TLS support. This plugin not only does in-flight
212encryption, but also performs client authorization based on the given
213endpoint's certificate chain. For example, a FoundationDB server may be
214configured to only accept client connections over TLS, where the client TLS
215certificate is from organization *Acme Co* in the
216*Research and Development* unit.
217
218Configuring TLS with FoundationDB is done using the
219{option}`services.foundationdb.tls` options in order to control the
220peer verification string, as well as the certificate and its private key.
221
222Note that the certificate and its private key must be accessible to the
223FoundationDB user account that the server runs under. These files are also
224NOT managed by NixOS, as putting them into the store may reveal private
225information.
226
227After you have a key and certificate file in place, it is not enough to
228simply set the NixOS module options -- you must also configure the
229{command}`fdb.cluster` file to specify that a given set of
230coordinators use TLS. This is as simple as adding the suffix
231{command}`:tls` to your cluster coordinator configuration, after the
232port number. For example, assuming you have a coordinator on localhost with
233the default configuration, simply specifying:
234
235```
236XXXXXX:XXXXXX@127.0.0.1:4500:tls
237```
238
239will configure all clients and server processes to use TLS from now on.
240
241## Backups and Disaster Recovery {#module-services-foundationdb-disaster-recovery}
242
243The usual rules for doing FoundationDB backups apply on NixOS as written in
244the FoundationDB manual. However, one important difference is the security
245profile for NixOS: by default, the {command}`foundationdb` systemd
246unit uses *Linux namespaces* to restrict write access to
247the system, except for the log directory, data directory, and the
248{command}`/etc/foundationdb/` directory. This is enforced by default
249and cannot be disabled.
250
251However, a side effect of this is that the {command}`fdbbackup`
252command doesn't work properly for local filesystem backups: FoundationDB
253uses a server process alongside the database processes to perform backups
254and copy the backups to the filesystem. As a result, this process is put
255under the restricted namespaces above: the backup process can only write to
256a limited number of paths.
257
258In order to allow flexible backup locations on local disks, the FoundationDB
259NixOS module supports a
260{option}`services.foundationdb.extraReadWritePaths` option. This
261option takes a list of paths, and adds them to the systemd unit, allowing
262the processes inside the service to write (and read) the specified
263directories.
264
265For example, to create backups in {command}`/opt/fdb-backups`, first
266set up the paths in the module options:
267
268```
269services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
270```
271
272Restart the FoundationDB service, and it will now be able to write to this
273directory (even if it does not yet exist.) Note: this path
274*must* exist before restarting the unit. Otherwise,
275systemd will not include it in the private FoundationDB namespace (and it
276will not add it dynamically at runtime).
277
278You can now perform a backup:
279
280```ShellSession
281$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
282$ sudo -u foundationdb fdbbackup status -t default
283```
284
285## Known limitations {#module-services-foundationdb-limitations}
286
287The FoundationDB setup for NixOS should currently be considered beta.
288FoundationDB is not new software, but the NixOS compilation and integration
289has only undergone fairly basic testing of all the available functionality.
290
291 - There is no way to specify individual parameters for individual
292 {command}`fdbserver` processes. Currently, all server processes
293 inherit all the global {command}`fdbmonitor` settings.
294 - Ruby bindings are not currently installed.
295 - Go bindings are not currently installed.
296
297## Options {#module-services-foundationdb-options}
298
299NixOS's FoundationDB module allows you to configure all of the most relevant
300configuration options for {command}`fdbmonitor`, matching it quite
301closely. A complete list of options for the FoundationDB module may be found
302[here](#opt-services.foundationdb.enable). You should
303also read the FoundationDB documentation as well.
304
305## Full documentation {#module-services-foundationdb-full-docs}
306
307FoundationDB is a complex piece of software, and requires careful
308administration to properly use. Full documentation for administration can be
309found here: <https://apple.github.io/foundationdb/>.