The KKK Cluster

Bugger all of that. This documentation is terrible. I’ve redone all of this in Debian and will publish better playbooks. I’ll let you know when that’s done.

Anyway, the crap I wrote half drunkenly installing a k8s cluster over a couple of nights:

I dropped a self-built Kubernetes cluster in favor of K3S. I now have it all configured with Flux.

When I fix my build issues, I’m hoping that I remember to relink it here.

–

Another ammendment. I picked this up again and rewrote the Playbook for Debian because I’m done with CentOS: https://gitlab.com/hxst/ansibles/kubernetes

–

Basic Theories

Let’s get your head around how this whole thing works. Firstly, there’s an awesome comic made by Scott McCloud that can be seen here. This will help you wrap your brain around how powerful it is, however you’ll likely just stare at a blank terminal, paralyzed under the weight of freedom, unable to progress the first time that you try to pick it up.

You can transfer a little bit of your hypervisor knowledge forward onto Kubernetes. At a basic level, you will be creating three hosts which will run a bunch of containers. I’m just going to assume that you know what a container is by this point.

The KKK suite helps us delegate which host will run which container, and how each container will work. It works just fine with Docker, so a lot of the images propping Docker up are transferrable into a Kubernetes cluster. Firstly, you’re going to need to configure your three starter hosts, configure one as a master (giggity) and the other two as slaves.

The idea behind what we want to do is have three machines configured to run containers, and let the KKK suite control what runs where. We want to configure things so that it doesn’t matter what host the container runs on, but I’ll also be running you through how to use labels to assign certain containers to specific hosts.

Configuring your Hosts (CentOS)

I swear, I’m getting de-ja-vu on this documentation.

I haven’t tried this out on Debian hosts yet, although I probably should. I wrote an Ansible role that will setup the host for us. The steps boil down to:

1. Ensure all hosts can communicate with each other over hostname (I just used a hosts file)
2. Configure Docker and Kubernetes repos
3. Configure kernel for k8s
4. Configure Docker to use systemd for it’s cgroup driver instead of systemd
5. Disable swap
6. Run a system update
7. Configure the firewall for the k8s plane
8. Install & Configure the KKK
9. Ensure that it is all running
10. Register the worker nodes with the master.

The steps to do this are accurately documented in the role, so I recommend reading through that if you’re not sure.

The link to the Ansible role: https://github.com/surprise-automation/ansible-first-k8s-cluster

1. Ensure Host Communication

Personally, I just did the following:

$ cat << EOF >> /etc/hosts
10.0.0.2 ed
10.0.0.3 edd
10.0.0.4 eddie
EOF

You can handle this with a properly configured DNS server though.

2. Configure Docker and Kubernetes Repos

The below is solely to configure the repos, as per steps above.

Docker

You can check the official installation guide here: https://docs.docker.com/engine/install/centos/

If you’re running on a fresh system, here’s the copypasta:

yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine
yum -y install yum-utils
yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
# yum -y install docker-ce docker-ce-cli containerd.io
# systemctl enable --now docker

Kubernetes

To install Kubernetes using official repos, check out the official installation guide here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management

Again, on a fresh system:

cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
# yum --disableexcludes=kubernetes -y install kubectl kubelet kubeadm 

The “disable excludes” flag is included because we have excluded the KKK trio from the kubernetes repo. There’s a warning on the kubeadm install page which explains this however the same warning isn’t present for kubectl and kubelet:

Warning: These instructions exclude all Kubernetes packages from any system upgrades. This is because kubeadm and Kubernetes require special attention to upgrade.

If you review the playbook that I have posted, you’ll see that we also include the trio in our .repo file and disableexcludes when installing in the roles tasks/main.yml file.

3. Configure Kernel for k8s

Your kernel should include the bridge module by default and you should be pretty aware if you have manually disabled it, or if it’s a feature of your distro. Regardless, RedHat flavors have the bridge module installed by default however you still need to the bridge to send packets to iptables prior to sending them to your containers.

My understanding on this is foggy, and I welcome ammendments via email, however I’ve sourced my information from the libvirt wiki.

I believe this comes from the fact that Docker/Kubernetes will leverage and control your hosts iptables installation to route traffic around. As such, traffic needs to pass through the iptables firewall in order to get to the containers correctly. The way to do this is pretty easy; drop the following into /etc/sysctl.d/k8s.conf:

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

You additionally need to enable the br_netfilter module for your kernel. Direct from Oracle, “This module is required to enable transparent masquerading and to facilitate Virtual Extensible LAN (VxLAN) traffic for communication between Kubernetes pods across the cluster. [source]”.

Create the file /etc/modules-load.d/k8s.conf and populate with:

br_netfilter

I’m fuzzy on loading kernels as it’s been a while since I ran Gentoo or Arch, but at this point a reboot wouldn’t hurt you. You can hot-load the kernel module with modprobe if you don’t want to reboot, but you’ll still need the above to load on boot:

modprobe br_netfilter

4. Configure Docker to use systemd for it’s cgroup driver instead of systemd

This is really important. If you skip this like I did, you’ll run into dramas. I did not document the error I received, however if you run into any cgroup driver errors - and you will if you don’t do this - then this is the solution.

cat << EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

5. Disable Swap

To permanent disable swaps on your system, comment out or delete the lines in /etc/fstab pertaining to swap. A swap line will look like this:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx swap                    swap    defaults        0 0

This will not take effect until a reboot, or maybe a mount -a however I have not tested the latter. To avoid a reboot at this point but still immediately disable swap, run the following to disable all swapping on the system:

swapoff -a

6. Run a system update

Naturally, you’ll want to run a full system update. You can do this at any point but I prefer to do it before installing KKK.

yum -y update

7. Configure the firewall for the k8s plane

The final piece of the puzzle is the firewall. You need to open all the management ports that Kubernetes uses.

Here’s some firewalld copy pasta for your enjoyment:

firewall-cmd --permanent --zone=public --add-port=179/tcp
firewall-cmd --permanent --zone=public --add-port=6443/tcp
firewall-cmd --permanent --zone=public --add-port=2379/tcp
firewall-cmd --permanent --zone=public --add-port=2380/tcp
firewall-cmd --permanent --zone=public --add-port=10250/tcp
firewall-cmd --permanent --zone=public --add-port=10251/tcp
firewall-cmd --permanent --zone=public --add-port=10252/tcp
firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
firewall-cmd --reload

8. Install & Configure the KKK

Finally, the moment we have all been waiting for. Installing KKK.

Install Kubeadm, Kubelet & Kubectl

yum --disableexcludes=kubernetes -y install kubelet kubeadm kubectl

Again, note that we are disabling excludes on the kubernetes repo.

Configuration

I have added installing the network prior to creating the control plane as it makes logical sense but it’s entirely theoretical. It goes against the docs but if I recall my experience correctly, I think it should have been done in this order.

If you run the network install prior to any control plane initialization and run into errors, please contact me.

Install Network: Project Calico

You need to install your network, which is a weird concept to me personally. I opt for Project Calico as it’s easy to get started with.

If you read their official quickstart guide, you will issues. They don’t tell you that you need to edit their manifest prior to ingesting with kubectl.

The best way to do this and use a custom CIDR is to download the manifest, make your edits, then ingest it:

curl https://docs.projectcalico.org/manifests/calico.yaml -O calico.yaml
sed -i 's/# - name: CALICO_IPV4POOL_CIDR/ - name: CALICO_IPV4POOL_CIDR/' calico.yaml
sed -i 's/#   value: "192.168.0.0/16"/   value: "10.8.0.0/16"/' calico.yaml

You can then apply it with kubectl:

kubectl apply -f calico.yaml

To check if it’s applying to all your nodes:

kubectl get pods -n kube-system

You should eventually see the STATUS turn to Running

Create Control Plane

On the host that is to be the master, run the following:

kubeadm init --pod-network-cidr=10.8.0.0/16

Note the join command it gives you, as it’ll make your life easier when you join the workers:

kubeadm join <masters ip>:6443 --token <token> \
        --discovery-token-ca-cert-hash <some hash>

9. Ensure that it is all running

To verify that everything is talking, you can run the following to get details of all pods:

kubectl get pods

10. Register the worker nodes with the master.

When you build your master node, you’ll have been given a command for binding worker nodes to the master. All that you need to do is run this command on the worker node, presuming that it can connect to the master, and let kubeadm do it’s thing.

You can check pod status via:

kubectl get pods

Your join command will look something like this:

kubeadm join <masters ip>:6443 --token <token> \
        --discovery-token-ca-cert-hash <some hash>

Run the above on the workers to join them to the master. You can run the following to get there join status:

kubectl get pods -n kube-system

Extras

Something that you should be aware of; kubectl requires that you have ~/.kube/config created and configured in order for kubectl to function correctly.

kubectl looks at ~/.kube/config to know how to connect to the api. If this is not configured, you’ll get a no response from localhost:8080 type of message, which appears to be the default port kubectl uses.

When running things as root, like the total badass that I am:

mkdir ~/.kube
ln -s /etc/kubernetes/admin.conf ~/.kube/config

If you’re running as a general user, I’d recommend just copying it:

[root@host ~]# mkdir /home/user/.kube/
[root@host ~]# cp /etc/kubernetes/admin.conf /home/user/.kube/config
[root@host ~]# chown -R user:group /home/user/.kube
[user@host ~]# kubectl get pods -n kube-system
NAME                          READY   STATUS    RESTARTS   AGE
coredns-78fcd69978-k5sbq      0/1     Pending   0          95m
coredns-78fcd69978-xzqhv      0/1     Pending   0          95m
etcd-edd                      1/1     Running   1          95m
kube-apiserver-edd            1/1     Running   1          95m
kube-controller-manager-edd   1/1     Running   1          95m
kube-proxy-82xhn              1/1     Running   0          56m
kube-proxy-8t4kg              1/1     Running   0          95m
kube-proxy-dz6hp              1/1     Running   0          56m
kube-scheduler-edd            1/1     Running   1          95m
[user@host ~]# kubectl get nodes
NAME    STATUS     ROLES                  AGE    VERSION
edd     NotReady   control-plane,master   105m   v1.22.1

That’s it. Thanks for coming to my TED Talk.

Upon review after doing this a few times now, I can definitely say that I had issues which I didn’t know to address. I remember after this, I had major issues getting CoreDNS to work and it was largely due to my CNI. I didn’t get the time to work out why it failed, but I’ll be re-addressing that very soon ;)