The vGeek Blog

A few months ago I began my journey to learn Kubernetes, the reason being that, besides it being a controversial technology, it will soon became a “must know” piece of technology for old VMware admins like myself. Project Pacific is a clear indicator that Kubernetes will become a first class citizen in vSphere infrastructure some time in 2020.

I took the Linux Foundation LFS458 Training and I am studying to pass the CKA – Certified Kubernetes Administrator exam. This exam is not based on questions that can be easily answered to with some prior cramming efforts, but it is lab based, with tasks to be completed in a real environment and with the clock ticking fast. So, real hands on experience is needed.

The LFS458 Training came with a Lab book and a test environment already set up on Google Cloud. Now, I would like to go through all those labs once again at my pace, but that environment is gone and I wanted to set up an equivalent one on my own that I can use (and destroy and rebuild, if needed) at will.

When I took the course last summer, it was based on Kubernetes 1.14.1, but that is already an old release and the latest stable one is now based on the 1.16.x branch, so obviously I wanted a lab environment that reflected that. I also wanted to base the lab on the latest Ubuntu LTS release (18.0 Bionic) but I learned the hard way that it is better to stick to the previous one (16.0 Xenial), since I came across a lot of issues (especially with CNI network plugins) when I tried to set up the cluster with Bionic. Note that – at the moment of writing – there are no Ubuntu Bionic (18.04) official K8s packages available, the latest released being those for Xenial (16.04). The situation might evolve quickly, so be on the lookout for announcements.

Another difference from the official LFS458 lab setup is that I decided to use Weave Net instead of Calico as it seems to be a more popular pod networking solution.

After many attempts, I came up with a working combination of components and this guide is based on:

• Ubuntu Xenial Server 16.04.6
• Docker 18.09.7
• Kubernetes 1.16.3
• Weave Net 2.6.0

Feel free to experiment with different combinations but be aware that your mileage may vary, and you will be alone out there, so be warned.

NOTE valid as of 27 Dec 2021: This post was written in December 2019 and all the components that are listed above have been updated. I have checked the procedure and I confirm it works with no modifications with the below BoM:

• Ubuntu Focal Server 20.04.3
• Docker 20.10.7
• Kubernetes 1.23.1
• Weave Net 2.8.1

Please note that Kubernetes has deprecated Docker as a supported container runtime since v 1.20 and will completely remove support very shortly, so this guide will not be updated but most likely replaced with another one based on a different container engine. Use of Docker container images will not be impacted as they are based on the OCI standards followed by any other supported container runtime.

This lab environment is only slightly better than Minikube, in the sense that it is a two nodes cluster with one master and one single worker. As you will see later, both nodes will be able to run non-infrastructure pods as the scheduling taint will be removed from the master. Nothing prevents you to deploy a more canonical setup with 1x master and 2x workers if you have the resources, that is up to you and the second worker can be easily added with the kubeadm join command.

I set up two Ubuntu Xenial VMs in my homelab ESXi NUC (2 vCPUs, 4 GB of RAM, 24 GB vHDD each) but any other platform will work, from VirtualBox on your laptop to EC2 instances in AWS. I performed a clean installation, assigned a static IP to both VMs and fully updated them.

Now let’s go for the step-by-step guide.

From now on, until explicitly advised, we need to perform the same steps on both nodes (basically installing Docker and kubeadm) so follow me.

Become root with:

$ sudo -i

Then install Docker from the Xenial repositories:

# apt-get install -y docker.io

In my previous, unsuccessful attempts I learned a couple of things about Docker. First, we need to ensure that Docker is using the correct cgroup driver as Kubernetes “prefers” the systemd one, while Docker defaults to cfgroups. Having two cgroup drivers co-existing at the same time is is not advisable, so before moving on, make sure that Docker is set to use the systemd cgroup driver as explained below.

The reasoning behind this is well explained in this article on the Kubernetes website. Below are the two steps required to force the systemd cgroups driver, first setting up the daemon, and then reloading it:

# cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

# systemctl daemon-reload
# systemctl restart docker

Second, I decided to not go for the latest version of Docker available at the time of writing (19.03.5 in Dec 2019) because the latest release supported with Kubernetes 1.14+ is 18.09. You might want to risk going for a more recent Docker release, but Kubernetes will throw a warning during the kubeadm init and I don’t really know what the consequences would be. Better safe than sorry.

As of today, the latest Docker version available in Xenial is 18.09.7, so that would work fine without the need of explicitly selecting this version nor marking the .deb packages on hold to avoid fortuitous upgrades (doing so won’t hurt, though):

Moving forward, let’s check the status of the Docker service after these changes:

# service docker status

and the version of Docker installed:

# docker --version

Before beginning the installation of K8s, it is advisable to disable swap:

# swapoff -a

The above command only works until the server is rebooted. Make it permanent by commenting the swap line in your /etc/fstab file. Failing to do so might generate issues when connecting to your API server (been there, done that). If you too come across the same issue that happened to me after a reboot (The connection to the server <host>:6443 was refused – did you specify the right host or port?), that is easily fixed in Ubuntu using the steps in this Kubernetes community post. Commenting fstab at this stage will prevent this from happening later.

Now it is time to configure the Kubernetes apt repositories in Ubuntu, create a new apt repository file for Kubernetes:

# vim /etc/apt/sources.list.d/kubernetes.list

and add this line and this line only:

deb http://apt.kubernetes.io/ kubernetes-xenial main

then import the GPG key:

# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add

Proceed with update the apt database:

# apt-get update

and with the installation of the latest kubeadm, kubelet and kubectl packages:

# apt-get install -y kubeadm kubelet kubectl

An older version can be installed by passing it as an option, e.g.:

# apt-get install -y kubeadm=1.14.1-00 kubelet=1.14.1-00 kubectl=1.14.1-00

Finally, ensure that the kube* packages are marked so that they are not accidentally upgraded whenever OS system patches are applied, as this might break your cluster:

# apt-mark hold kubelet kubeadm kubectl

The common preparation part is complete, now we focus on the master node only, but before doing that, it is important to decide which Pod Networking solution we will install later as this might affect the deployment strategy.

I preferred to go for WeaveWorks Weave Net for its simplicity, but you can use anything else like Flannel or Calico.

Weave takes care of selecting and applying a CIDR block for the pod network and segmenting it across nodes (by default it chooses 10.32.0.0/12, but it is configurable if needed). With Calico, the network is defined in the calico.yaml file and this subnet must be explicitly passed as a parameter to kubeadm during the init with the --pod-network-cidr <Pod CIDR> option.

In our case, since we are using Weave Net, the master can be bootstrapped with a simple command (a copy of the output is saved for reference):

# kubeadm init | tee kubeadm-init.out

At the end of the init process, if there are no errors (there should not), you should take note of two important pieces of info: the token that you will need to join the workers, and the commands to be run as a standard user to enable kubectl access to the cluster API server.

Note that the token will be valid only for 24 hrs, but a new one can be generated at any time with the kubeadm token create command if it expires in the meanwhile. Let’s then configure kubectl authentication by performing the following steps.

Exit the root session and become your standard user again:

# exit

Copy the skeleton kubeadm config file and place it under your home:

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Now test the connection to the API server:

$ kubectl get nodes

Connection should be OK but the master should show up as Not Ready.

This is expected because we need to implement the Pod Networking solution first. Let’s also check the pods in the kube-system namespace:

$ kubectl -n=kube-system get pods

Ignore the errors (CoreDNS pods in Pending state) for now: this is normal as the cluster is not yet operational.

Let’s install Weave. This is as simple as downloading a YAML definition file and applying it (long line is split for readability):

$ kubectl apply -f \
"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version| base64 | tr -d '\n')"

Again, check the status of pods and other resources, the pods should now show up all as running:

$ kubectl -n=kube-system get all

and, finally, the node should be in Ready status:

$ kubectl get nodes

Time to move to the other node and join it to the cluster as a worker. You should have already performed all the steps related to the installation of Docker, kubeadm, kubelet and kubectl, so let’s proceed with the joining:

# kubeadm join 192.168.25.41:6443 --token <YOUR TOKEN> \
--discovery-token-ca-cert-hash sha256:<YOUR TOKEN CA HASH>

As soon as the joining operation completes, go back to your master and check the status of the nodes, both should be in Ready state:

$ kubectl get nodes

and then check that the daemon sets for weave and kube-proxy are working for both nodes:

$ kubectl -n=kube-system get all -o wide

Finally, we have to allow the master server to run non-infrastructure pods: the master node started up tainted for security and performance reasons. This is absolutely a must-be for any production cluster but, since this is a small two nodes cluster to be used for learning purposes only, we can go ahead and un-taint the master. An alternative, if you have enough resources, would be to deploy a second worker node: doing so there would be no need to un-taint the master.

Allowing the master to run non-system pods can be set with the command:

$ kubectl taint nodes --all node-role.kubernetes.io/master-

(mind the dash at the end of the line, that’s the one doing the magic).

Running kubectl describe node <master_node> before and after the untaint shows the difference:

Your small CKA cluster is finally ready! Happy labbing and see you for the (hopefully) next K8s blogpost!

If you like this post, please share it!