From Zero to K3S on Proxmox LXC: Part 1 - Creating the Cluster

This is the first of a series of posts describing how to bootstrap a Kubernetes cluster on Proxmox using LXC containers.

By the end of the series, the aim is to have a fully working Kubernetes (K3S) install including MetalLB load balancer, NGINX ingress controller and an Istio service mesh. I’ll also have some sample applications installed for good measure.

But first, a promise — all parts of the series are already published here. I’ve been bitten too many times by the tantalising missing “part 2”.

The series will be:

• Part 1 (this post) — setting up K3S on multiple LXC VMs hosted in a Proxmox server
• Part 2 — Configuring and testing MetalLB and Nginx Ingress
• Part 3 — Installing Kubernetes Dashboard and setting with an Ingress
• Part 4 — Installing and testing Istio Service Mesh

Yet another K3S on Proxmox post ? Why ?!?

Over the past few months, I’ve made various attempts at setting up a usable K8S cluster for my own experiments and learning. Most of these were abandoned due to lack of time or random unexplained failures but I recently found enough time to go through it all again and this time, I won !

There’s a tonne of great articles already out there describing how to do this, and I’ve read a lot of them (many will be linked here). They’re all very informative but I found they often lacked some crucial information or only covered part of the process. Everyone is trying to do different things with their set up so of course, I figured one more attempt couldn’t hurt.

So, I have questions…

Why do I need a Kubernetes cluster ?

To be honest, that’s a fair question. The best answer I can come up with is… because it’s there. At work, I’ve used large K8S clusters in Production for longer than I care to think about but they’ve always been managed by someone else. That’s a great thing for getting stuff done™ but it leaves me with an embarrassing lack of practical knowledge in how things work. Nothing beats DIY in terms of practical learning.

In short, I wanted an environment in which I could experiment and learn. I run a reasonably complex home network in order to confuse, delight and torment my family and I had grand plans to port all my random docker-compose scripts and VMs into a K8S cluster. Tbh, I suspect that’s overkill and will probably never happen but there’s always hope.

Why use Proxmox and LXC ? Why not VMs ?

I’ve been on a bit of a journey with how I run my home services over the last 20 years or so but I settled on Proxmox as a virtualisation server a few years back and it’s been pretty much rock solid since. Where possible, I try and use LXC containers rather than full-fledged VMs as they tend to be less resource hungry and are generally easier to use (imho). That said, installing K3S does require a few tweaks to the LXC containers  —  nothing too complicated, mind.

What is K3S ?

This is much better described elsewhere…

K3s is a lightweight Kubernetes distribution created by Rancher Labs, and it is fully certified by the Cloud Native Computing Foundation (CNCF). K3s is highly available and production-ready. It has a very small binary size and very low resource requirements.

In simple terms, K3s is Kubernetes with bloat stripped out and a different backing datastore. That said, it is important to note that K3s is not a fork, as it doesn’t change any of the core Kubernetes functionalities and remains close to stock Kubernetes.
(source: https://traefik.io/glossary/k3s-explained/)

Why K3S ? Why not <insert mini-k8s favourite here> ?

There are plenty of lightweight kubernetes solutions to choose from and I’ve tried a few of them in the past (KIND, minikube, microk8s etc). They work pretty well and I’m sure some of them would be suitable here too. I only started playing with K3S recently after seeing how well it’s documented and have been pretty impressed so far. YMMV but it seems like a good place to start.

Cluster Topology

I want to set up a fairly straightforward cluster that fits well into my home network. I tend to use DHCP for most hosts on the network but for the kubernetes nodes I’m creating, I’ll keep things simple with static IP addresses.

I want to be able to access services deployed in the cluster from elsewhere on my network, using both HTTP and HTTPS. There are various ways to achieve this but the most appropriate for me is to set up an Nginx Ingress controller with a statically assigned IP range allocated in my network’s subnet. I’ll set up corresponding host names for those IPs in my DNS server.

I use dnsmasq as a DHCP and DNS server. It’s a handy way to configure static host names for arbitrary IP addresses that every machine on my network can resolve — this will be useful when I set up a new ingress hostname for a test service (see part 2). Other DNS servers are available but with dnsmasq, it’s as simple as adding a new entry in the /etc/hosts file on my DNS server and reloading config (ref).

For the purposes of this guide, I’m going to use 3 fixed IP addresses and a hardcoded domain name (cluster.mydomain.org) that only resolves inside my network:

• 192.168.1.40 — kube-master.cluster.mydomain.org (the main k3s node)
• 192.168.1.41 — kube-worker-1.cluster.mydomain.org (a k3s worker node)
• 192.168.1.50 — an ingress IP address pool with only a single IP in it. I’ve added mappings in the dnsmasq /etc/hosts file for every application hostname / service that I want to expose with an ingress

Creating the Kubernetes nodes

I’m going to create two new containers in Proxmox (Create CT) on which I can install K3S —  I used a Ubuntu 20.04.1 LTS template, other distros are available.

The settings I used for both K3S nodes:

• 2 CPU cores, 32GB disk space, 4GB RAM & No Swap
• Privileged container
• Static IP addresses (I added these IPs to my DNS server’s static records as above)

Configuring the container

After starting the new container, I updated packages and installed curl :

apt update && apt upgrade && apt install curl

Adding a user for running K3S

I chose to run everything as a user called kubernetes so I had to create that with adduser kubernetes. 

I used visudo to add kubernetes ALL=(ALL) NOPASSWD:ALL to the sudoers file so that the new user can sudo commands without entering a password.

From my laptop, I can now ssh kubernetes@kube-master 

Now, I need to modify the LXC container with some extra permissions required by K3S. To do that, first I need to stop the container so I can edit /etc/pve/lxc/<container_id>.conf on the Proxmox host itself.

I added the following to the LXC config file for the new container ID (source): 

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"

K3S also needs /dev/kmsg to exist inside the container but LXC won’t offer that by default so the container needs to be tweaked a little (source).

After starting the container and logging in again, I’ll add an /etc/rc.local script:

echo '#!/bin/sh -e
ln -s /dev/console /dev/kmsg
mount - make-rshared /' > /etc/rc.local
chmod +x /etc/rc.local
reboot

Now, take a backup…

It’s a good idea to take a backup of the new container at this point. It’ll come in handy if I ever need to reset to a baseline. Since I’m also going to create a K3S worker node shortly, I can also use this backup to create a new container without having to repeat the steps above.

Installing K3S

Okay, so I’ve created a clean LXC container called kube-master with IP address 192.168.1.40 and now I want to install K3S on it. As always, there are various ways to do this but I found the easiest was to use k3sup.

First, k3sup must be installed on my laptop...

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/
k3sup -- help

Now, I can use k3sup from my laptop to install onto the kube-master node:

k3sup install --ip 192.168.1.40 \
   --k3s-extra-args="-disable traefik -disable servicelb" \
   --user kubernetes

So what happened ?

Running: k3sup install
2024/01/04 18:21:36 192.168.1.40
Public IP: 192.168.1.40
[INFO] Finding release for channel stable
[INFO] Using v1.28.5+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service -> /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Result: [INFO] Finding release for channel stable
[INFO] Using v1.28.5+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.28.5+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service -> /etc/systemd/system/k3s.service.
Saving file to: /Users/andrew/kubernetes/kubeconfig
# Test your cluster with:
export KUBECONFIG=/Users/andrew/kubernetes/kubeconfig
kubectl config use-context default
kubectl get node -o wide

Well, that was painless.

So what next ? 

K3S is now up and running on my master node but I want to connect from my laptop to verify it’s working. I’ll need to configure my laptop to talk to the new cluster...

k3supgenerates a kubeconfig file in the current directory when it’s finished installing so I can use this with kubectl . That’s either a case of copying this file over the top of .kube/config or setting the KUBECONFIGenvironment variable,

export KUBECONFIG=`pwd`/kubeconfig
kubectl get nodes

And I see the magic result...

NAME        STATUS ROLES                AGE  VERSION
kube-master Ready  control-plane,master 2m3s v1.28.5+k3s1

Adding a worker node

So I have a single node cluster up and running now but where’s the fun in that ? I want to add a worker node which means I’ll need another LXC container.

Since I took a backup of the master node after I configured it and before I installed K3S on it, I can use that backup to create a replica with a different hostname & IP address.

I made sure to,

• Select ‘Unique’ restore so that the replica gets a new MAC address
• Override the hostname to ‘kube-worker-1’
• NOT select ‘start after restore’ 

I didn’t start the container immediately because I need to change the configured static IP address to match the new hostname (kube-worker-1 — 192.168.1.41/24) but that’s literally the only change I needed to make for this new container before joining it to the K3S cluster with,

k3sup join --ip 192.168.1.41 --server-ip 192.168.1.40 --user kubernetes

When that completes, kubectl get nodes now returns,

NAME           STATUS   ROLES                  AGE   VERSION
kube-master    Ready    control-plane,master   15m   v1.28.5+k3s1
kube-worker-1  Ready    <none>                 12s   v1.28.5+k3s1

And… I’m done

Well, okay not exactly done. 

I now have a vanilla multi-node Kubernetes cluster running in a couple of LXC containers and accessible from my laptop. It’s got nothing deployed inside it yet but that’s easily fixed… see part 2.