pyrox.dev
1# Kubernetes {#sec-kubernetes}
2
3The NixOS Kubernetes module is a collective term for a handful of
4individual submodules implementing the Kubernetes cluster components.
5
6There are generally two ways of enabling Kubernetes on NixOS. One way is
7to enable and configure cluster components appropriately by hand:
8
9```nix
10{
11 services.kubernetes = {
12 apiserver.enable = true;
13 controllerManager.enable = true;
14 scheduler.enable = true;
15 addonManager.enable = true;
16 proxy.enable = true;
17 flannel.enable = true;
18 };
19}
20```
21
22Another way is to assign cluster roles ("master" and/or "node") to
23the host. This enables apiserver, controllerManager, scheduler,
24addonManager, kube-proxy and etcd:
25
26```nix
27{ services.kubernetes.roles = [ "master" ]; }
28```
29
30While this will enable the kubelet and kube-proxy only:
31
32```nix
33{ services.kubernetes.roles = [ "node" ]; }
34```
35
36Assigning both the master and node roles is usable if you want a single
37node Kubernetes cluster for dev or testing purposes:
38
39```nix
40{
41 services.kubernetes.roles = [
42 "master"
43 "node"
44 ];
45}
46```
47
48Note: Assigning either role will also default both
49[](#opt-services.kubernetes.flannel.enable)
50and [](#opt-services.kubernetes.easyCerts)
51to true. This sets up flannel as CNI and activates automatic PKI bootstrapping.
52
53::: {.note}
54It is mandatory to configure:
55[](#opt-services.kubernetes.masterAddress).
56The masterAddress must be resolvable and routable by all cluster nodes.
57In single node clusters, this can be set to `localhost`.
58:::
59
60Role-based access control (RBAC) authorization mode is enabled by
61default. This means that anonymous requests to the apiserver secure port
62will expectedly cause a permission denied error. All cluster components
63must therefore be configured with x509 certificates for two-way tls
64communication. The x509 certificate subject section determines the roles
65and permissions granted by the apiserver to perform clusterwide or
66namespaced operations. See also: [ Using RBAC
67Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).
68
69The NixOS kubernetes module provides an option for automatic certificate
70bootstrapping and configuration,
71[](#opt-services.kubernetes.easyCerts).
72The PKI bootstrapping process involves setting up a certificate authority (CA)
73daemon (cfssl) on the kubernetes master node. cfssl generates a CA-cert
74for the cluster, and uses the CA-cert for signing subordinate certs issued
75to each of the cluster components. Subsequently, the certmgr daemon monitors
76active certificates and renews them when needed. For single node Kubernetes
77clusters, setting [](#opt-services.kubernetes.easyCerts)
78= true is sufficient and no further action is required. For joining extra node
79machines to an existing cluster on the other hand, establishing initial
80trust is mandatory.
81
82To add new nodes to the cluster: On any (non-master) cluster node where
83[](#opt-services.kubernetes.easyCerts)
84is enabled, the helper script `nixos-kubernetes-node-join` is available on PATH.
85Given a token on stdin, it will copy the token to the kubernetes secrets directory
86and restart the certmgr service. As requested certificates are issued, the
87script will restart kubernetes cluster components as needed for them to
88pick up new keypairs.
89
90::: {.note}
91Multi-master (HA) clusters are not supported by the easyCerts module.
92:::
93
94In order to interact with an RBAC-enabled cluster as an administrator,
95one needs to have cluster-admin privileges. By default, when easyCerts
96is enabled, a cluster-admin kubeconfig file is generated and linked into
97`/etc/kubernetes/cluster-admin.kubeconfig` as determined by
98[](#opt-services.kubernetes.pki.etcClusterAdminKubeconfig).
99`export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig` will make
100kubectl use this kubeconfig to access and authenticate the cluster. The
101cluster-admin kubeconfig references an auto-generated keypair owned by
102root. Thus, only root on the kubernetes master may obtain cluster-admin
103rights by means of this file.