pyrox.dev
1# FoundationDB {#module-services-foundationdb}
2
3*Source:* {file}`modules/services/databases/foundationdb.nix`
4
5*Upstream documentation:* <https://apple.github.io/foundationdb/>
6
7*Maintainer:* Austin Seipp
8
9*Available version(s):* 7.1.x
10
11FoundationDB (or "FDB") is an open source, distributed, transactional
12key-value store.
13
14## Configuring and basic setup {#module-services-foundationdb-configuring}
15
16To enable FoundationDB, add the following to your
17{file}`configuration.nix`:
18```nix
19{
20 services.foundationdb.enable = true;
21 services.foundationdb.package = pkgs.foundationdb73; # FoundationDB 7.3.x
22}
23```
24
25The {option}`services.foundationdb.package` option is required, and
26must always be specified. Due to the fact FoundationDB network protocols and
27on-disk storage formats may change between (major) versions, and upgrades
28must be explicitly handled by the user, you must always manually specify
29this yourself so that the NixOS module will use the proper version. Note
30that minor, bugfix releases are always compatible.
31
32After running {command}`nixos-rebuild`, you can verify whether
33FoundationDB is running by executing {command}`fdbcli` (which is
34added to {option}`environment.systemPackages`):
35```ShellSession
36$ sudo -u foundationdb fdbcli
37Using cluster file `/etc/foundationdb/fdb.cluster'.
38
39The database is available.
40
41Welcome to the fdbcli. For help, type `help'.
42fdb> status
43
44Using cluster file `/etc/foundationdb/fdb.cluster'.
45
46Configuration:
47 Redundancy mode - single
48 Storage engine - memory
49 Coordinators - 1
50
51Cluster:
52 FoundationDB processes - 1
53 Machines - 1
54 Memory availability - 5.4 GB per process on machine with least available
55 Fault Tolerance - 0 machines
56 Server time - 04/20/18 15:21:14
57
58...
59
60fdb>
61```
62
63You can also write programs using the available client libraries. For
64example, the following Python program can be run in order to grab the
65cluster status, as a quick example. (This example uses
66{command}`nix-shell` shebang support to automatically supply the
67necessary Python modules).
68```ShellSession
69a@link> cat fdb-status.py
70#! /usr/bin/env nix-shell
71#! nix-shell -i python -p python pythonPackages.foundationdb73
72
73import fdb
74import json
75
76def main():
77 fdb.api_version(520)
78 db = fdb.open()
79
80 @fdb.transactional
81 def get_status(tr):
82 return str(tr['\xff\xff/status/json'])
83
84 obj = json.loads(get_status(db))
85 print('FoundationDB available: %s' % obj['client']['database_status']['available'])
86
87if __name__ == "__main__":
88 main()
89a@link> chmod +x fdb-status.py
90a@link> ./fdb-status.py
91FoundationDB available: True
92a@link>
93```
94
95FoundationDB is run under the {command}`foundationdb` user and group
96by default, but this may be changed in the NixOS configuration. The systemd
97unit {command}`foundationdb.service` controls the
98{command}`fdbmonitor` process.
99
100By default, the NixOS module for FoundationDB creates a single SSD-storage
101based database for development and basic usage. This storage engine is
102designed for SSDs and will perform poorly on HDDs; however it can handle far
103more data than the alternative "memory" engine and is a better default
104choice for most deployments. (Note that you can change the storage backend
105on-the-fly for a given FoundationDB cluster using
106{command}`fdbcli`.)
107
108Furthermore, only 1 server process and 1 backup agent are started in the
109default configuration. See below for more on scaling to increase this.
110
111FoundationDB stores all data for all server processes under
112{file}`/var/lib/foundationdb`. You can override this using
113{option}`services.foundationdb.dataDir`, e.g.
114```nix
115{ services.foundationdb.dataDir = "/data/fdb"; }
116```
117
118Similarly, logs are stored under {file}`/var/log/foundationdb`
119by default, and there is a corresponding
120{option}`services.foundationdb.logDir` as well.
121
122## Scaling processes and backup agents {#module-services-foundationdb-scaling}
123
124Scaling the number of server processes is quite easy; simply specify
125{option}`services.foundationdb.serverProcesses` to be the number of
126FoundationDB worker processes that should be started on the machine.
127
128FoundationDB worker processes typically require 4GB of RAM per-process at
129minimum for good performance, so this option is set to 1 by default since
130the maximum amount of RAM is unknown. You're advised to abide by this
131restriction, so pick a number of processes so that each has 4GB or more.
132
133A similar option exists in order to scale backup agent processes,
134{option}`services.foundationdb.backupProcesses`. Backup agents are
135not as performance/RAM sensitive, so feel free to experiment with the number
136of available backup processes.
137
138## Clustering {#module-services-foundationdb-clustering}
139
140FoundationDB on NixOS works similarly to other Linux systems, so this
141section will be brief. Please refer to the full FoundationDB documentation
142for more on clustering.
143
144FoundationDB organizes clusters using a set of
145*coordinators*, which are just specially-designated
146worker processes. By default, every installation of FoundationDB on NixOS
147will start as its own individual cluster, with a single coordinator: the
148first worker process on {command}`localhost`.
149
150Coordinators are specified globally using the
151{command}`/etc/foundationdb/fdb.cluster` file, which all servers and
152client applications will use to find and join coordinators. Note that this
153file *can not* be managed by NixOS so easily:
154FoundationDB is designed so that it will rewrite the file at runtime for all
155clients and nodes when cluster coordinators change, with clients
156transparently handling this without intervention. It is fundamentally a
157mutable file, and you should not try to manage it in any way in NixOS.
158
159When dealing with a cluster, there are two main things you want to do:
160
161 - Add a node to the cluster for storage/compute.
162 - Promote an ordinary worker to a coordinator.
163
164A node must already be a member of the cluster in order to properly be
165promoted to a coordinator, so you must always add it first if you wish to
166promote it.
167
168To add a machine to a FoundationDB cluster:
169
170 - Choose one of the servers to start as the initial coordinator.
171 - Copy the {command}`/etc/foundationdb/fdb.cluster` file from this
172 server to all the other servers. Restart FoundationDB on all of these
173 other servers, so they join the cluster.
174 - All of these servers are now connected and working together in the
175 cluster, under the chosen coordinator.
176
177At this point, you can add as many nodes as you want by just repeating the
178above steps. By default there will still be a single coordinator: you can
179use {command}`fdbcli` to change this and add new coordinators.
180
181As a convenience, FoundationDB can automatically assign coordinators based
182on the redundancy mode you wish to achieve for the cluster. Once all the
183nodes have been joined, simply set the replication policy, and then issue
184the {command}`coordinators auto` command
185
186For example, assuming we have 3 nodes available, we can enable double
187redundancy mode, then auto-select coordinators. For double redundancy, 3
188coordinators is ideal: therefore FoundationDB will make
189*every* node a coordinator automatically:
190
191```ShellSession
192fdbcli> configure double ssd
193fdbcli> coordinators auto
194```
195
196This will transparently update all the servers within seconds, and
197appropriately rewrite the {command}`fdb.cluster` file, as well as
198informing all client processes to do the same.
199
200## Client connectivity {#module-services-foundationdb-connectivity}
201
202By default, all clients must use the current {command}`fdb.cluster`
203file to access a given FoundationDB cluster. This file is located by default
204in {command}`/etc/foundationdb/fdb.cluster` on all machines with the
205FoundationDB service enabled, so you may copy the active one from your
206cluster to a new node in order to connect, if it is not part of the cluster.
207
208## Client authorization and TLS {#module-services-foundationdb-authorization}
209
210By default, any user who can connect to a FoundationDB process with the
211correct cluster configuration can access anything. FoundationDB uses a
212pluggable design to transport security, and out of the box it supports a
213LibreSSL-based plugin for TLS support. This plugin not only does in-flight
214encryption, but also performs client authorization based on the given
215endpoint's certificate chain. For example, a FoundationDB server may be
216configured to only accept client connections over TLS, where the client TLS
217certificate is from organization *Acme Co* in the
218*Research and Development* unit.
219
220Configuring TLS with FoundationDB is done using the
221{option}`services.foundationdb.tls` options in order to control the
222peer verification string, as well as the certificate and its private key.
223
224Note that the certificate and its private key must be accessible to the
225FoundationDB user account that the server runs under. These files are also
226NOT managed by NixOS, as putting them into the store may reveal private
227information.
228
229After you have a key and certificate file in place, it is not enough to
230simply set the NixOS module options -- you must also configure the
231{command}`fdb.cluster` file to specify that a given set of
232coordinators use TLS. This is as simple as adding the suffix
233{command}`:tls` to your cluster coordinator configuration, after the
234port number. For example, assuming you have a coordinator on localhost with
235the default configuration, simply specifying:
236
237```
238XXXXXX:XXXXXX@127.0.0.1:4500:tls
239```
240
241will configure all clients and server processes to use TLS from now on.
242
243## Backups and Disaster Recovery {#module-services-foundationdb-disaster-recovery}
244
245The usual rules for doing FoundationDB backups apply on NixOS as written in
246the FoundationDB manual. However, one important difference is the security
247profile for NixOS: by default, the {command}`foundationdb` systemd
248unit uses *Linux namespaces* to restrict write access to
249the system, except for the log directory, data directory, and the
250{command}`/etc/foundationdb/` directory. This is enforced by default
251and cannot be disabled.
252
253However, a side effect of this is that the {command}`fdbbackup`
254command doesn't work properly for local filesystem backups: FoundationDB
255uses a server process alongside the database processes to perform backups
256and copy the backups to the filesystem. As a result, this process is put
257under the restricted namespaces above: the backup process can only write to
258a limited number of paths.
259
260In order to allow flexible backup locations on local disks, the FoundationDB
261NixOS module supports a
262{option}`services.foundationdb.extraReadWritePaths` option. This
263option takes a list of paths, and adds them to the systemd unit, allowing
264the processes inside the service to write (and read) the specified
265directories.
266
267For example, to create backups in {command}`/opt/fdb-backups`, first
268set up the paths in the module options:
269
270```nix
271{ services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ]; }
272```
273
274Restart the FoundationDB service, and it will now be able to write to this
275directory (even if it does not yet exist.) Note: this path
276*must* exist before restarting the unit. Otherwise,
277systemd will not include it in the private FoundationDB namespace (and it
278will not add it dynamically at runtime).
279
280You can now perform a backup:
281
282```ShellSession
283$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
284$ sudo -u foundationdb fdbbackup status -t default
285```
286
287## Known limitations {#module-services-foundationdb-limitations}
288
289The FoundationDB setup for NixOS should currently be considered beta.
290FoundationDB is not new software, but the NixOS compilation and integration
291has only undergone fairly basic testing of all the available functionality.
292
293 - There is no way to specify individual parameters for individual
294 {command}`fdbserver` processes. Currently, all server processes
295 inherit all the global {command}`fdbmonitor` settings.
296 - Ruby bindings are not currently installed.
297 - Go bindings are not currently installed.
298
299## Options {#module-services-foundationdb-options}
300
301NixOS's FoundationDB module allows you to configure all of the most relevant
302configuration options for {command}`fdbmonitor`, matching it quite
303closely. A complete list of options for the FoundationDB module may be found
304[here](#opt-services.foundationdb.enable). You should
305also read the FoundationDB documentation as well.
306
307## Full documentation {#module-services-foundationdb-full-docs}
308
309FoundationDB is a complex piece of software, and requires careful
310administration to properly use. Full documentation for administration can be
311found here: <https://apple.github.io/foundationdb/>.