pyrox.dev
1<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="sec-kubernetes">
2 <title>Kubernetes</title>
3 <para>
4 The NixOS Kubernetes module is a collective term for a handful of
5 individual submodules implementing the Kubernetes cluster
6 components.
7 </para>
8 <para>
9 There are generally two ways of enabling Kubernetes on NixOS. One
10 way is to enable and configure cluster components appropriately by
11 hand:
12 </para>
13 <programlisting language="bash">
14services.kubernetes = {
15 apiserver.enable = true;
16 controllerManager.enable = true;
17 scheduler.enable = true;
18 addonManager.enable = true;
19 proxy.enable = true;
20 flannel.enable = true;
21};
22</programlisting>
23 <para>
24 Another way is to assign cluster roles ("master" and/or
25 "node") to the host. This enables apiserver,
26 controllerManager, scheduler, addonManager, kube-proxy and etcd:
27 </para>
28 <programlisting language="bash">
29services.kubernetes.roles = [ "master" ];
30</programlisting>
31 <para>
32 While this will enable the kubelet and kube-proxy only:
33 </para>
34 <programlisting language="bash">
35services.kubernetes.roles = [ "node" ];
36</programlisting>
37 <para>
38 Assigning both the master and node roles is usable if you want a
39 single node Kubernetes cluster for dev or testing purposes:
40 </para>
41 <programlisting language="bash">
42services.kubernetes.roles = [ "master" "node" ];
43</programlisting>
44 <para>
45 Note: Assigning either role will also default both
46 <xref linkend="opt-services.kubernetes.flannel.enable" /> and
47 <xref linkend="opt-services.kubernetes.easyCerts" /> to true. This
48 sets up flannel as CNI and activates automatic PKI bootstrapping.
49 </para>
50 <para>
51 As of kubernetes 1.10.X it has been deprecated to open
52 non-tls-enabled ports on kubernetes components. Thus, from NixOS
53 19.03 all plain HTTP ports have been disabled by default. While
54 opening insecure ports is still possible, it is recommended not to
55 bind these to other interfaces than loopback. To re-enable the
56 insecure port on the apiserver, see options:
57 <xref linkend="opt-services.kubernetes.apiserver.insecurePort" />
58 and
59 <xref linkend="opt-services.kubernetes.apiserver.insecureBindAddress" />
60 </para>
61 <note>
62 <para>
63 As of NixOS 19.03, it is mandatory to configure:
64 <xref linkend="opt-services.kubernetes.masterAddress" />. The
65 masterAddress must be resolveable and routeable by all cluster
66 nodes. In single node clusters, this can be set to
67 <literal>localhost</literal>.
68 </para>
69 </note>
70 <para>
71 Role-based access control (RBAC) authorization mode is enabled by
72 default. This means that anonymous requests to the apiserver secure
73 port will expectedly cause a permission denied error. All cluster
74 components must therefore be configured with x509 certificates for
75 two-way tls communication. The x509 certificate subject section
76 determines the roles and permissions granted by the apiserver to
77 perform clusterwide or namespaced operations. See also:
78 <link xlink:href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">
79 Using RBAC Authorization</link>.
80 </para>
81 <para>
82 The NixOS kubernetes module provides an option for automatic
83 certificate bootstrapping and configuration,
84 <xref linkend="opt-services.kubernetes.easyCerts" />. The PKI
85 bootstrapping process involves setting up a certificate authority
86 (CA) daemon (cfssl) on the kubernetes master node. cfssl generates a
87 CA-cert for the cluster, and uses the CA-cert for signing
88 subordinate certs issued to each of the cluster components.
89 Subsequently, the certmgr daemon monitors active certificates and
90 renews them when needed. For single node Kubernetes clusters,
91 setting <xref linkend="opt-services.kubernetes.easyCerts" /> = true
92 is sufficient and no further action is required. For joining extra
93 node machines to an existing cluster on the other hand, establishing
94 initial trust is mandatory.
95 </para>
96 <para>
97 To add new nodes to the cluster: On any (non-master) cluster node
98 where <xref linkend="opt-services.kubernetes.easyCerts" /> is
99 enabled, the helper script
100 <literal>nixos-kubernetes-node-join</literal> is available on PATH.
101 Given a token on stdin, it will copy the token to the kubernetes
102 secrets directory and restart the certmgr service. As requested
103 certificates are issued, the script will restart kubernetes cluster
104 components as needed for them to pick up new keypairs.
105 </para>
106 <note>
107 <para>
108 Multi-master (HA) clusters are not supported by the easyCerts
109 module.
110 </para>
111 </note>
112 <para>
113 In order to interact with an RBAC-enabled cluster as an
114 administrator, one needs to have cluster-admin privileges. By
115 default, when easyCerts is enabled, a cluster-admin kubeconfig file
116 is generated and linked into
117 <literal>/etc/kubernetes/cluster-admin.kubeconfig</literal> as
118 determined by
119 <xref linkend="opt-services.kubernetes.pki.etcClusterAdminKubeconfig" />.
120 <literal>export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig</literal>
121 will make kubectl use this kubeconfig to access and authenticate the
122 cluster. The cluster-admin kubeconfig references an auto-generated
123 keypair owned by root. Thus, only root on the kubernetes master may
124 obtain cluster-admin rights by means of this file.
125 </para>
126</chapter>