Skip to main content

K3s Homelab Deployment Guide

K3s Homelab Deployment Guide

Deploy a lightweight, production-ready Kubernetes cluster using K3s + Cilium + Helm + nginx LB. Designed for homelab environments on Proxmox VE or bare-metal Debian nodes.

Architecture

  • Master runs the K3s control plane, Cilium operator, Hubble UI, and core DNS
  • Workers run Cilium agents + Envoy sidecars
  • nginx LB does TCP stream proxying for K3s API (6443) and round-robin to Cilium Ingress NodePorts

Prerequisites

  • Master: 1-2 vCPU, 2-4 GB RAM
  • Workers: 1-2 vCPU, 2-4 GB RAM each
  • LB: 1 vCPU, 512 MB - 1 GB RAM
  • OS: Debian 12/13 (cloud-init image)
  • Network: All nodes reachable on same L2/L3 subnet

Step 1 — Prerequisites on all nodes

swapoff -a
sed -i '/ swap /d' /etc/fstab
modprobe br_netfilter
modprobe overlay
echo 'br_netfilter' > /etc/modules-load.d/k8s.conf
echo 'overlay' >> /etc/modules-load.d/k8s.conf
echo 'net.bridge.bridge-nf-call-iptables=1' > /etc/sysctl.d/k8s.conf
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.d/k8s.conf
sysctl --system

Step 2 — Install K3s Master

export K3S_TOKEN="your-secure-token"

curl -sfL https://get.k3s.io | K3S_TOKEN=${K3S_TOKEN} sh -s - server --cluster-init --disable=traefik --flannel-backend=none --disable-network-policy --node-name=kubemaster

Why disable Traefik and Flannel? Cilium replaces both the default K3s CNI (flannel) and the default ingress controller (Traefik). Disabling them prevents conflicts.
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
kubectl get nodes

Step 3 — Install Helm

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add cilium https://helm.cilium.io/
helm repo update

Step 4 — Install Cilium (CNI + Ingress + Hubble)

helm install cilium cilium/cilium --version 1.16.0 --namespace kube-system 
  --set kubeProxyReplacement=true 
  --set k8sServiceHost=10.28.38.27 
  --set k8sServicePort=6443 
  --set ingressController.enabled=true 
  --set ingressController.loadbalancerMode=dedicated 
  --set ingressController.default=true 
  --set ipam.mode=kubernetes 
  --set hubble.enabled=true 
  --set hubble.relay.enabled=true 
  --set hubble.ui.enabled=true 
  --set operator.replicas=1

Wait for Cilium pods:

watch kubectl get pods -n kube-system

Step 5 — Join Worker Nodes

On each worker:

export K3S_TOKEN="same-token"
export K3S_URL="https://10.28.38.27:6443"

curl -sfL https://get.k3s.io | K3S_TOKEN=${K3S_TOKEN} K3S_URL=${K3S_URL} sh -s - agent --node-name=kubeworker01

Verify from master:

kubectl get nodes -o wide

Step 6 — Configure nginx Load Balancer

Option A — TCP Stream (L4) — Simple, single ingress

stream {
    upstream k3s_api { server 10.28.38.27:6443; }
    server { listen 6443; proxy_pass k3s_api; }

upstream k8s_http { server 10.28.38.27:31014; server 10.28.38.28:31014; server 10.28.38.29:31014; } server { listen 80; proxy_pass k8s_http; } }

Option B — HTTP Reverse Proxy (L7) — Multi-domain, dedicated ingresses

http {
    upstream cuancouple_http {
        server 10.28.38.27:31460;
        server 10.28.38.28:31460;
        server 10.28.38.29:31460;
    }

upstream wordpress_http { server 10.28.38.27:32012; server 10.28.38.28:32012; server 10.28.38.29:32012; }

server { listen 80 default_server; server_name _; location / { proxy_pass http://cuancouple_http; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_http_version 1.1; proxy_set_header Connection ""; } }

server { listen 80; server_name dosys.my.id; location / { proxy_pass http://wordpress_http; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_http_version 1.1; proxy_set_header Connection ""; } } }

Step 7 — Verify

kubectl get nodes  # All Ready
kubectl get pods -n kube-system  # All Running
kubectl -n kube-system exec ds/cilium -- cilium status --brief  # OK

Container Image Build & Import

K3s uses containerd, not Docker. Build on master, import everywhere:

# Build
docker build -t myapp:latest .
docker save myapp:latest > /tmp/myapp.tar

Import on ALL nodes

for IP in 10.28.38.27 10.28.38.28 10.28.38.29; do scp /tmp/myapp.tar root@${IP}:/tmp/ ssh root@${IP} "ctr -n k8s.io images import /tmp/myapp.tar" done
Critical: Use ctr -n k8s.io images import — the k8s.io namespace is where K3s expects images. docker build alone is not enough.

Expose via Cloudflare Tunnel

curl -L -o cloudflared.deb https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
dpkg -i cloudflared.deb
cloudflared service install

Traffic flow: Internet → Cloudflare → cloudflared → localhost:80 → nginx → Cilium Ingress → Pods

Common Pitfalls

  • Cilium install fails with connection refused — Set export KUBECONFIG=~/.kube/config before helm install
  • Node stays NotReady — Wait 30-60s for Cilium agent init containers to finish
  • ImagePullBackOff — Images must be imported via ctr -n k8s.io, not just docker build
  • MySQL 8.0 crashes with x86-64-v2 error — Use mariadb:10.6 on older CPUs (i5-7500, Xeon E3 v5, etc.)
  • NFS mount hangs (ContainerCreating) — Install nfs-common on all K3s nodes
  • Cloudflare Tunnel hits wrong app — Ensure server_name in nginx matches Ingress host exactly
  • LoadBalancer stays pending — Expected in homelab; use NodePort directly
  • K3s master install hangs — Ensure swap is disabled and kernel modules loaded
  • Workers cannot reach master:6443 — Check firewall rules allowing TCP 6443
  • Dedicated ingress svclb pods stuck in Pending — Remove shared cilium-ingress service if using dedicated mode

No comments to display

Back to top