···
+
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-foundationdb">
+
<title>FoundationDB</title>
+
<emphasis>Source:</emphasis>
+
<filename>modules/services/databases/foundationdb.nix</filename>
+
<emphasis>Upstream documentation:</emphasis>
+
<link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>
+
<emphasis>Maintainer:</emphasis> Austin Seipp
+
<emphasis>Available version(s):</emphasis> 5.1.x, 5.2.x, 6.0.x
+
FoundationDB (or "FDB") is an open source, distributed,
+
transactional key-value store.
+
<section xml:id="module-services-foundationdb-configuring">
+
<title>Configuring and basic setup</title>
+
To enable FoundationDB, add the following to your
+
<filename>configuration.nix</filename>:
+
services.foundationdb.enable = true;
+
services.foundationdb.package = pkgs.foundationdb52; # FoundationDB 5.2.x
+
The <option>services.foundationdb.package</option> option is
+
required, and must always be specified. Due to the fact
+
FoundationDB network protocols and on-disk storage formats may
+
change between (major) versions, and upgrades must be explicitly
+
handled by the user, you must always manually specify this
+
yourself so that the NixOS module will use the proper version.
+
Note that minor, bugfix releases are always compatible.
+
After running <command>nixos-rebuild</command>, you can verify
+
whether FoundationDB is running by executing
+
<command>fdbcli</command> (which is added to
+
<option>environment.systemPackages</option>):
+
$ sudo -u foundationdb fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.
The database is available.
Welcome to the fdbcli. For help, type `help'.
Using cluster file `/etc/foundationdb/fdb.cluster'.
···
+
You can also write programs using the available client libraries.
+
For example, the following Python program can be run in order to
+
grab the cluster status, as a quick example. (This example uses
+
<command>nix-shell</command> shebang support to automatically
+
supply the necessary Python modules).
+
a@link> cat fdb-status.py
#! /usr/bin/env nix-shell
#! nix-shell -i python -p python pythonPackages.foundationdb52
···
obj = json.loads(get_status(db))
print('FoundationDB available: %s' % obj['client']['database_status']['available'])
+
if __name__ == "__main__":
+
a@link> chmod +x fdb-status.py
+
a@link> ./fdb-status.py
FoundationDB available: True
+
FoundationDB is run under the <command>foundationdb</command> user
+
and group by default, but this may be changed in the NixOS
+
configuration. The systemd unit
+
<command>foundationdb.service</command> controls the
+
<command>fdbmonitor</command> process.
+
By default, the NixOS module for FoundationDB creates a single
+
SSD-storage based database for development and basic usage. This
+
storage engine is designed for SSDs and will perform poorly on
+
HDDs; however it can handle far more data than the alternative
+
"memory" engine and is a better default choice for most
+
deployments. (Note that you can change the storage backend
+
on-the-fly for a given FoundationDB cluster using
+
<command>fdbcli</command>.)
+
Furthermore, only 1 server process and 1 backup agent are started
+
in the default configuration. See below for more on scaling to
+
FoundationDB stores all data for all server processes under
+
<filename>/var/lib/foundationdb</filename>. You can override this
+
using <option>services.foundationdb.dataDir</option>, e.g.
+
services.foundationdb.dataDir = "/data/fdb";
+
Similarly, logs are stored under
+
<filename>/var/log/foundationdb</filename> by default, and there
+
is a corresponding <option>services.foundationdb.logDir</option>
+
<section xml:id="module-services-foundationdb-scaling">
+
<title>Scaling processes and backup agents</title>
+
Scaling the number of server processes is quite easy; simply
+
specify <option>services.foundationdb.serverProcesses</option> to
+
be the number of FoundationDB worker processes that should be
+
started on the machine.
+
FoundationDB worker processes typically require 4GB of RAM
+
per-process at minimum for good performance, so this option is set
+
to 1 by default since the maximum amount of RAM is unknown. You're
+
advised to abide by this restriction, so pick a number of
+
processes so that each has 4GB or more.
+
A similar option exists in order to scale backup agent processes,
+
<option>services.foundationdb.backupProcesses</option>. Backup
+
agents are not as performance/RAM sensitive, so feel free to
+
experiment with the number of available backup processes.
+
<section xml:id="module-services-foundationdb-clustering">
+
<title>Clustering</title>
+
FoundationDB on NixOS works similarly to other Linux systems, so
+
this section will be brief. Please refer to the full FoundationDB
+
documentation for more on clustering.
+
FoundationDB organizes clusters using a set of
+
<emphasis>coordinators</emphasis>, which are just
+
specially-designated worker processes. By default, every
+
installation of FoundationDB on NixOS will start as its own
+
individual cluster, with a single coordinator: the first worker
+
process on <command>localhost</command>.
+
Coordinators are specified globally using the
+
<command>/etc/foundationdb/fdb.cluster</command> file, which all
+
servers and client applications will use to find and join
+
coordinators. Note that this file <emphasis>can not</emphasis> be
+
managed by NixOS so easily: FoundationDB is designed so that it
+
will rewrite the file at runtime for all clients and nodes when
+
cluster coordinators change, with clients transparently handling
+
this without intervention. It is fundamentally a mutable file, and
+
you should not try to manage it in any way in NixOS.
+
When dealing with a cluster, there are two main things you want to
+
<itemizedlist spacing="compact">
+
Add a node to the cluster for storage/compute.
+
Promote an ordinary worker to a coordinator.
+
A node must already be a member of the cluster in order to
+
properly be promoted to a coordinator, so you must always add it
+
first if you wish to promote it.
+
To add a machine to a FoundationDB cluster:
+
<itemizedlist spacing="compact">
+
Choose one of the servers to start as the initial coordinator.
+
Copy the <command>/etc/foundationdb/fdb.cluster</command> file
+
from this server to all the other servers. Restart
+
FoundationDB on all of these other servers, so they join the
+
All of these servers are now connected and working together in
+
the cluster, under the chosen coordinator.
+
At this point, you can add as many nodes as you want by just
+
repeating the above steps. By default there will still be a single
+
coordinator: you can use <command>fdbcli</command> to change this
+
and add new coordinators.
+
As a convenience, FoundationDB can automatically assign
+
coordinators based on the redundancy mode you wish to achieve for
+
the cluster. Once all the nodes have been joined, simply set the
+
replication policy, and then issue the
+
<command>coordinators auto</command> command
+
For example, assuming we have 3 nodes available, we can enable
+
double redundancy mode, then auto-select coordinators. For double
+
redundancy, 3 coordinators is ideal: therefore FoundationDB will
+
make <emphasis>every</emphasis> node a coordinator automatically:
+
fdbcli> configure double ssd
+
fdbcli> coordinators auto
+
This will transparently update all the servers within seconds, and
+
appropriately rewrite the <command>fdb.cluster</command> file, as
+
well as informing all client processes to do the same.
+
<section xml:id="module-services-foundationdb-connectivity">
+
<title>Client connectivity</title>
+
By default, all clients must use the current
+
<command>fdb.cluster</command> file to access a given FoundationDB
+
cluster. This file is located by default in
+
<command>/etc/foundationdb/fdb.cluster</command> on all machines
+
with the FoundationDB service enabled, so you may copy the active
+
one from your cluster to a new node in order to connect, if it is
+
not part of the cluster.
+
<section xml:id="module-services-foundationdb-authorization">
+
<title>Client authorization and TLS</title>
+
By default, any user who can connect to a FoundationDB process
+
with the correct cluster configuration can access anything.
+
FoundationDB uses a pluggable design to transport security, and
+
out of the box it supports a LibreSSL-based plugin for TLS
+
support. This plugin not only does in-flight encryption, but also
+
performs client authorization based on the given endpoint's
+
certificate chain. For example, a FoundationDB server may be
+
configured to only accept client connections over TLS, where the
+
client TLS certificate is from organization <emphasis>Acme
+
Co</emphasis> in the <emphasis>Research and Development</emphasis>
+
Configuring TLS with FoundationDB is done using the
+
<option>services.foundationdb.tls</option> options in order to
+
control the peer verification string, as well as the certificate
+
Note that the certificate and its private key must be accessible
+
to the FoundationDB user account that the server runs under. These
+
files are also NOT managed by NixOS, as putting them into the
+
store may reveal private information.
+
After you have a key and certificate file in place, it is not
+
enough to simply set the NixOS module options -- you must also
+
configure the <command>fdb.cluster</command> file to specify that
+
a given set of coordinators use TLS. This is as simple as adding
+
the suffix <command>:tls</command> to your cluster coordinator
+
configuration, after the port number. For example, assuming you
+
have a coordinator on localhost with the default configuration,
XXXXXX:XXXXXX@127.0.0.1:4500:tls
+
will configure all clients and server processes to use TLS from
+
<section xml:id="module-services-foundationdb-disaster-recovery">
+
<title>Backups and Disaster Recovery</title>
+
The usual rules for doing FoundationDB backups apply on NixOS as
+
written in the FoundationDB manual. However, one important
+
difference is the security profile for NixOS: by default, the
+
<command>foundationdb</command> systemd unit uses <emphasis>Linux
+
namespaces</emphasis> to restrict write access to the system,
+
except for the log directory, data directory, and the
+
<command>/etc/foundationdb/</command> directory. This is enforced
+
by default and cannot be disabled.
+
However, a side effect of this is that the
+
<command>fdbbackup</command> command doesn't work properly for
+
local filesystem backups: FoundationDB uses a server process
+
alongside the database processes to perform backups and copy the
+
backups to the filesystem. As a result, this process is put under
+
the restricted namespaces above: the backup process can only write
+
to a limited number of paths.
+
In order to allow flexible backup locations on local disks, the
+
FoundationDB NixOS module supports a
+
<option>services.foundationdb.extraReadWritePaths</option> option.
+
This option takes a list of paths, and adds them to the systemd
+
unit, allowing the processes inside the service to write (and
+
read) the specified directories.
+
For example, to create backups in
+
<command>/opt/fdb-backups</command>, first set up the paths in the
+
services.foundationdb.extraReadWritePaths = [ "/opt/fdb-backups" ];
+
Restart the FoundationDB service, and it will now be able to write
+
to this directory (even if it does not yet exist.) Note: this path
+
<emphasis>must</emphasis> exist before restarting the unit.
+
Otherwise, systemd will not include it in the private FoundationDB
+
namespace (and it will not add it dynamically at runtime).
+
You can now perform a backup:
+
$ sudo -u foundationdb fdbbackup start -t default -d file:///opt/fdb-backups
+
$ sudo -u foundationdb fdbbackup status -t default
+
<section xml:id="module-services-foundationdb-limitations">
+
<title>Known limitations</title>
+
The FoundationDB setup for NixOS should currently be considered
+
beta. FoundationDB is not new software, but the NixOS compilation
+
and integration has only undergone fairly basic testing of all the
+
available functionality.
+
<itemizedlist spacing="compact">
+
There is no way to specify individual parameters for
+
individual <command>fdbserver</command> processes. Currently,
+
all server processes inherit all the global
+
<command>fdbmonitor</command> settings.
+
Ruby bindings are not currently installed.
+
Go bindings are not currently installed.
+
<section xml:id="module-services-foundationdb-options">
+
NixOS's FoundationDB module allows you to configure all of the
+
most relevant configuration options for
+
<command>fdbmonitor</command>, matching it quite closely. A
+
complete list of options for the FoundationDB module may be found
+
<link linkend="opt-services.foundationdb.enable">here</link>. You
+
should also read the FoundationDB documentation as well.
+
<section xml:id="module-services-foundationdb-full-docs">
+
<title>Full documentation</title>
+
FoundationDB is a complex piece of software, and requires careful
+
administration to properly use. Full documentation for
+
administration can be found here:
+
<link xlink:href="https://apple.github.io/foundationdb/" role="uri">https://apple.github.io/foundationdb/</link>.
pyrox.dev