From Zero to K3S on Proxmox LXC: Part 2 - Ingress

In part 1 of this series, I described how to create a K3S cluster using Proxmox LXC nodes. but at the end of that post, I had a cluster up and running but with some limitations - services running inside the cluster aren't currently accessible from the rest of my network.

In order to make services available outside the cluster, I'll need to set up an Ingress controller and a load balancer - I'm going to use ingress-nginx and MetalLB.

What's an Ingress controller and why do I need one?

An Ingress controller is a specialized load balancer for Kubernetes environments. An Ingress controller abstracts away the complexity of Kubernetes application traffic routing and provides a bridge between Kubernetes services and external ones.

Kubernetes Ingress controllers:
- Accept traffic from outside the Kubernetes platform, and load balance it to pods running inside the platform
- Are configured using the Kubernetes API to deploy objects called “Ingress Resources”
- Monitor the pods running in Kubernetes and automatically update the load‑balancing rules when pods are added or removed from a service

(source: nginx.com)

I want to be able to deploy an HTTP service in my new K3S cluster and access it from anywhere on my home network. There are various ways to do this with Kubernetes but the most flexible is to set up an ingress.

Why Nginx ?

K3S comes with Traefik as the default Ingress controller but I've chosen to use NGINX instead. There's no strong reason for this other than a bit of cargo-culting. Most of the articles I followed to get my cluster stood up recommending adopting NGINX Ingress over Traefik as it's more standard. I don't know if that's strictly relevant but since I tried both and I could only get Nginx working in my set up, I'm happy to stick with it.

Why MetalLB ?

In cloud environments, a Kubernetes ingress can use out-of-the-box load balancer instances to assign new IP addresses when a new ingress resource (service address) is created.

Since my home network doesn't have an on-demand load balancer, I'll need to provide the ingress controller with a software equivalent.

MetalLB provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.

In a Layer 2 configuration mode of MetalLB together with the NGINX Ingress controller, one node attracts all the traffic for the ingress-nginx Service IP.

(Source: kubernetes.github.io)

Let's install some stuff

MetalLB

I chose to use an installation from manifest:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.12/config/manifests/metallb-native.yaml

That appeared to be successful:

namespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
configmap/metallb-excludel2 created
secret/webhook-server-cert created
service/webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created

I can also verify that MetalLB pods are running in the new metallb-system namespace:

╰─ kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-786f9df989-x5rpx   1/1     Running   0          2m
speaker-xfnf7                 1/1     Running   0          2m
speaker-qxmxb                 1/1     Running   0          2m

Configuring an IP address pool

Now, for the first bit of actual kubernetes configuration I've had to do in this series.

I need to tell MetalLB what IP address range it can use, and how to advertise it to other hosts on my network.

The manifest for this configuration is as follows:

# Tell MetalLB what IP address range it can use
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  # Only using a single address for my network
  - 192.168.1.50-192.168.1.50

---

# Tell MetalLB to advertise it's IP addresses on the local network.
# This means it will answer ARP requests for the IP addresses in the pool 
# without needing to configure those IP addresses on the nodes themselves.
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertisement
  namespace: metallb-system

After applying the manifest, I can see the MetalLB resources were created:

╰─ kubectl apply -f 01_metallb-ipaddr-pool-with-l2advertisement.yaml
ipaddresspool.metallb.io/default-pool created
l2advertisement.metallb.io/l2-advertisement created

Installing ingress-nginx

The good news is that installing ingress-nginx is easy and well documented - the 'Bare Metal' installation is most applicable for my setup but this time, I'll use a Helm chart instead of a plain manifest file:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --wait

After about 30 seconds, I get some helpful output:

Release "ingress-nginx" does not exist. Installing it now.
NAME: ingress-nginx
LAST DEPLOYED: Fri Jan  5 14:58:35 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the load balancer IP to be available.
You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch'

And I can see the ingress is using the External IP from MetalLB 🎉:

╰─ kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE    SELECTOR
ingress-nginx-controller   LoadBalancer   10.43.29.231   192.168.1.50   80:32192/TCP,443:30782/TCP   3m2s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

Installing cert-manager

Okay, so we have an ingress controller working so I can expose services over HTTP. But I'd also like to be able to use HTTPS which means I'll need my Kubernetes cluster to have Certificate Manager installed.

This can also be installed with Helm:

# Add the Jetstack Helm repository if you haven't already done so
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install the cert-manager Helm chart
# This command also creates some custom resource definitions (CRDs)
# that allows configuring certificate manager via manifests
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.13.3 \
  --set installCRDs=true

Now, I can verify that cert-manager is running in it's new namespace:

kubectl -n cert-manager get pods
NAME                                      READY   STATUS    RESTARTS AGE
cert-manager-cainjector-d7f8b5464-rvtth   1/1     Running   0        15m
cert-manager-57688f5dc6-92jrg             1/1     Running   0        15m
cert-manager-webhook-58fd67545d-ntcjd     1/1     Running   0        15m

I still need to configure a ClusterIssuer (this is why I needed to install the cert-manager CRDs) using a manifest:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

Which results in...

kubectl apply -f 02_certmanager-clusterissuer.yaml
clusterissuer.cert-manager.io/selfsigned created

Ok, cool. But how do we know if it's all working ?

I'm going to install a basic application into K3S and expose it via HTTPs on a dedicated hostname to verify everything I've done so far is working together.

This will involve:

• Deploying a new deployment and service with a basic web server in a new namespace (testapp)
• Creating an HTTPS ingress and TLS certificate for testapp.cluster.mydomain.org
• Adding testapp.cluster.mydomain.org to my DNS server
• Verifying I can open https://testapp.cluster.mydomain.org from a browser

So first, I need to create the new namespace:

kubectl create namespace testapp

Next, I'll deploy a test application without the ingress:

# Define a deployment in the 'testapp' namesapce
# Uses a basic 'nginx' webserver image and listens on port 80
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: testapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

---

# Define a service for the 'nginx' deployment in the 'testapp' namespace
# Service should listen on HTTP 80 and HTTPS 443
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: testapp
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: https
  selector:
    app: nginx

Now, I'll create an Ingress resource and certificate for the new service:

# Create a certificate secret matching the TLS cert specified above 
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: testapp-cluster-mydomain-org-tls
  namespace: testapp
spec:
  commonName: testapp.cluster.mydomain.org
  secretName: testapp-cluster-mydomain-org-tls
  dnsNames:
    - testapp.cluster.mydomain.org
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

---

# Define an NGINX TLS ingress for the 'nginx-svc' service
# Uses the 'selfsigned' cluster issuer we set up earlier
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: testapp-ingress
  namespace: testapp
  annotations:
    cert-manager.io/cluster-issuer: selfsigned
    nginx.ingress.kubernetes.io/router.tls: "true"
spec:
  # Specifies use of the 'nginx' ingress controller
  ingressClassName: nginx
  rules:

  # Match requests for the service hostname 
  # 'testapp.cluster.mydomain.org' to this ingress service
  - host: testapp.cluster.mydomain.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-svc
            port:
              number: 80
  tls:
  - hosts:
    - testapp.cluster.mydomain.org
    secretName: testapp-cluster-mydomain-org-tls

The final piece of the puzzle is to ensure that my ingress hostname testapp.cluster.mydomain.org resolves to the IP address assigned by my load balancer (192.168.1.50).

In my case, this is as simple as adding an entry to the /etc/hosts file for my dnsmasq server and reloading:

127.0.0.1       localhost.localdomain localhost
192.168.1.40    kube-master
192.168.1.41    kube-worker-1

192.168.1.50    testapp.cluster.mydomain.org

Does it all work ?

If everything has worked so far, I should be able to open https://testapp.cluster.mydomain.org from my laptop browser.

The first time I open it, I'll get a warning because I've configured cert-manager to use a self-signed TLS certificate.

But if I accept that warning, I get the sweet, sweet nginx default page served by my test application inside the K3S cluster🎉:

Great, so what's next ?

Part 1 of this series covered getting a K3S cluster created.

Part 2 covered setting up a load balancer (MetalLB), an Ingress controller (Nginx) and a Certificate Issuer. Finally, I've verified that I can deploy a test application in the cluster and access it via HTTPS.

Next up, in Part 3, I'll install the Kubernetes Dashboard along with an ingress and access tokens so I can administer the cluster.