pyrox.dev
1<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-kubernetes">
2 <title>Kubernetes</title>
3 <para>
4 The NixOS Kubernetes module is a collective term for a handful of
5 individual submodules implementing the Kubernetes cluster
6 components.
7 </para>
8 <para>
9 There are generally two ways of enabling Kubernetes on NixOS. One
10 way is to enable and configure cluster components appropriately by
11 hand:
12 </para>
13 <programlisting language="bash">
14services.kubernetes = {
15 apiserver.enable = true;
16 controllerManager.enable = true;
17 scheduler.enable = true;
18 addonManager.enable = true;
19 proxy.enable = true;
20 flannel.enable = true;
21};
22</programlisting>
23 <para>
24 Another way is to assign cluster roles ("master" and/or
25 "node") to the host. This enables apiserver,
26 controllerManager, scheduler, addonManager, kube-proxy and etcd:
27 </para>
28 <programlisting language="bash">
29services.kubernetes.roles = [ "master" ];
30</programlisting>
31 <para>
32 While this will enable the kubelet and kube-proxy only:
33 </para>
34 <programlisting language="bash">
35services.kubernetes.roles = [ "node" ];
36</programlisting>
37 <para>
38 Assigning both the master and node roles is usable if you want a
39 single node Kubernetes cluster for dev or testing purposes:
40 </para>
41 <programlisting language="bash">
42services.kubernetes.roles = [ "master" "node" ];
43</programlisting>
44 <para>
45 Note: Assigning either role will also default both
46 <xref linkend="opt-services.kubernetes.flannel.enable" /> and
47 <xref linkend="opt-services.kubernetes.easyCerts" /> to true. This
48 sets up flannel as CNI and activates automatic PKI bootstrapping.
49 </para>
50 <note>
51 <para>
52 As of NixOS 19.03, it is mandatory to configure:
53 <xref linkend="opt-services.kubernetes.masterAddress" />. The
54 masterAddress must be resolveable and routeable by all cluster
55 nodes. In single node clusters, this can be set to
56 <literal>localhost</literal>.
57 </para>
58 </note>
59 <para>
60 Role-based access control (RBAC) authorization mode is enabled by
61 default. This means that anonymous requests to the apiserver secure
62 port will expectedly cause a permission denied error. All cluster
63 components must therefore be configured with x509 certificates for
64 two-way tls communication. The x509 certificate subject section
65 determines the roles and permissions granted by the apiserver to
66 perform clusterwide or namespaced operations. See also:
67 <link xlink:href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">
68 Using RBAC Authorization</link>.
69 </para>
70 <para>
71 The NixOS kubernetes module provides an option for automatic
72 certificate bootstrapping and configuration,
73 <xref linkend="opt-services.kubernetes.easyCerts" />. The PKI
74 bootstrapping process involves setting up a certificate authority
75 (CA) daemon (cfssl) on the kubernetes master node. cfssl generates a
76 CA-cert for the cluster, and uses the CA-cert for signing
77 subordinate certs issued to each of the cluster components.
78 Subsequently, the certmgr daemon monitors active certificates and
79 renews them when needed. For single node Kubernetes clusters,
80 setting <xref linkend="opt-services.kubernetes.easyCerts" /> = true
81 is sufficient and no further action is required. For joining extra
82 node machines to an existing cluster on the other hand, establishing
83 initial trust is mandatory.
84 </para>
85 <para>
86 To add new nodes to the cluster: On any (non-master) cluster node
87 where <xref linkend="opt-services.kubernetes.easyCerts" /> is
88 enabled, the helper script
89 <literal>nixos-kubernetes-node-join</literal> is available on PATH.
90 Given a token on stdin, it will copy the token to the kubernetes
91 secrets directory and restart the certmgr service. As requested
92 certificates are issued, the script will restart kubernetes cluster
93 components as needed for them to pick up new keypairs.
94 </para>
95 <note>
96 <para>
97 Multi-master (HA) clusters are not supported by the easyCerts
98 module.
99 </para>
100 </note>
101 <para>
102 In order to interact with an RBAC-enabled cluster as an
103 administrator, one needs to have cluster-admin privileges. By
104 default, when easyCerts is enabled, a cluster-admin kubeconfig file
105 is generated and linked into
106 <literal>/etc/kubernetes/cluster-admin.kubeconfig</literal> as
107 determined by
108 <xref linkend="opt-services.kubernetes.pki.etcClusterAdminKubeconfig" />.
109 <literal>export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig</literal>
110 will make kubectl use this kubeconfig to access and authenticate the
111 cluster. The cluster-admin kubeconfig references an auto-generated
112 keypair owned by root. Thus, only root on the kubernetes master may
113 obtain cluster-admin rights by means of this file.
114 </para>
115</chapter>