Skip to main content

Monitoring Per-Pod Kubernetes dengan Grafana Alloy (K3s Homelab)

Monitoring Per-Pod Kubernetes dengan Grafana Alloy

Dokumentasi implementasi pipeline monitoring per-pod di cluster K3s homelab (kubemaster 10.28.38.27, kubeworker01 .28, kubeworker02 .29) menggunakan Grafana Alloy sebagai collector, Prometheus sebagai TSDB, dan Grafana sebagai visualisasi. Deploy dilakukan tanggal 7 September 2026.

1. Ringkasan & Akses

  • Grafana UI: http://10.28.38.27:30300 (juga via node .28/.29) — user admin, password: lihat Secret grafana-admin di namespace monitoring: 
    kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d
  • Prometheus: http://10.28.38.27:30902 (NodePort, read-only)
  • Dashboard: folder Kubernetes → Kubernetes - Per Pod Monitoring (provisioned, uid k8s-pods-overview)
  • Namespace: monitoring

2. Arsitektur

        ┌───────────────────── setiap node K3s ─────────────────────┐
        │  DaemonSet: Grafana Alloy v1.19.2                         │
        │   ├─ scrape kubelet :10250/metrics/cadvisor  (per-pod)    │
        │   ├─ scrape kubelet :10250/metrics           (node/kubelet)│
        │   ├─ scrape kube-state-metrics :8080         (objek API)  │
        │   └─ scrape alloy self :12345                (health)     │
        └───────────────────────────┬───────────────────────────────┘
                                    │ prometheus.remote_write
                                    ▼
                   Prometheus v3.14.0  (:9090, PVC 15Gi, retention 30d)
                                    ▲
                                    │ query
                   Grafana 13.2.1 (:3000 → NodePort 30300)

Alloy berjalan sebagai DaemonSet sehingga setiap node men-scrape kubelet lokalnya (data cadvisor = metrik CPU/memory/network/filesystem per container/pod). Karena cluster belum punya TSDB, Prometheus ikut dideploy sebagai target remote_write. Kubelet tidak mengekspos read-only port 10255, jadi scrape dilakukan ke HTTPS 10250 memakai ServiceAccount token dengan RBAC nodes/metrics (kubelet webhook authorization mengecek izin ini lewat SubjectAccessReview).

3. Komponen & Versi

  • grafana/alloy:v1.19.2 — collector, mode DaemonSet
  • quay.io/prometheus/prometheus:v3.14.0 — TSDB, hanya menerima remote_write (tanpa scrape_configs)
  • registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 — metrik objek API server (pod phase, restarts, limits, dsb)
  • grafana/grafana:13.2.1 — visualisasi, datasource Prometheus provisioned
  • Storage: PVC local-path (prometheus-data 15Gi, grafana-data 2Gi)

4. File Manifest

Tersimpan di host automation: /root/k8s-monitoring/

00-namespace.yaml          → ns monitoring
10-alloy-rbac.yaml         → SA alloy + ClusterRole (nodes, nodes/metrics, pods, services) + binding
20-alloy-config.yaml       → ConfigMap config.alloy (River syntax)
30-alloy-daemonset.yaml    → DaemonSet alloy (+initContainer fix-data-perms)
40-kube-state-metrics.yaml → SA+RBAC+Deployment+Service KSM
50-prometheus.yaml         → Config, PVC, Deployment, Service NodePort 30902
60-grafana.yaml            → Secret admin, PVC, datasources, dashboard provider, Deployment, NodePort 30300
+ ConfigMap grafana-dashboards-k8s (dashboard JSON, dibuat via --from-file)

5. Deployment

kubectl apply -f 00-namespace.yaml -f 10-alloy-rbac.yaml -f 20-alloy-config.yaml \
  -f 30-alloy-daemonset.yaml -f 40-kube-state-metrics.yaml -f 50-prometheus.yaml -f 60-grafana.yaml

kubectl create configmap grafana-dashboards-k8s -n monitoring \
  --from-file=k8s-pods.json=/tmp/dashboard-k8s-pods.json --dry-run=client -o yaml | kubectl apply -f -

6. Bagian Penting Config Alloy (River)

// discovery semua node via API server
discovery.kubernetes "kube_nodes" { role = "node" }

// relabel ke path cadvisor + label node
discovery.relabel "kubelet_cadvisor" {
  targets = discovery.kubernetes.kube_nodes.targets
  rule { target_label = "__metrics_path__", replacement = "/metrics/cadvisor" }
  rule { target_label = "__scheme__",      replacement = "https" }
  rule { source_labels = ["__meta_kubernetes_node_name"], target_label = "node" }
}

// scrape dengan token SA (block syntax, bukan attribute!)
prometheus.scrape "kubelet_cadvisor" {
  job_name = "kubelet-cadvisor"
  targets  = discovery.relabel.kubelet_cadvisor.output
  scheme   = "https"
  authorization {
    type             = "Bearer"
    credentials_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  }
  tls_config { insecure_skip_verify = true }
  forward_to = [prometheus.remote_write.prom.receiver]
}

prometheus.remote_write "prom" {
  endpoint { url = "http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write" }
}

7. Dashboard

Dashboard custom k8s-pods-overview (folder: Kubernetes) berisi 14 panel:

  • Overview (stat): Node Ready, Pods Running, CPU cluster (cores), Memory cluster, Pods per Namespace
  • Per Pod (timeseries): CPU cores, memory working set, network rx/tx, CPU throttling, FS read/write, restarts/h, CPU vs limit (%)
  • Table: Pods Not Running (phase != Running/Succeeded)
  • Template variables: Namespace & Pod (multi-select, via label_values(kube_pod_info, namespace))

8. Verifikasi

# pods harus Running
kubectl -n monitoring get pods

# data masuk ke Prometheus
kubectl -n monitoring exec deploy/prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=count(container_cpu_usage_seconds_total)'

# semua job up (kubelet, kubelet-cadvisor, kube-state-metrics, alloy)
... /api/v1/query?query=up

# hasil terverifikasi saat deploy: 85 series cadvisor, 23 pod via KSM, 3/3 kubelet up

9. Pitfall yang Ditemukan (PENTING)

  • UID Alloy = 473, bukan 474. Salah set fsGroup/runAsUser → mkdir /var/lib/alloy/data: permission denied crashloop. Pakai runAsUser/runAsGroup/fsGroup: 473 + initContainer busybox chmod sebagai pengaman.
  • Syntax River: authorization dan tls_config di prometheus.scrape adalah BLOCK, bukan attribute. tls_config = { ... } → error "tls_config" must be a block, but is used as an attribute. Yang benar: tls_config { ... } tanpa koma. http_client_config tidak tersedia di prometheus.scrape v1.19.
  • Prometheus butuh flag --web.enable-remote-write-receiver. Tanpa ini remote_write dari Alloy gagal dengan 404: remote write receiver needs to be enabled.
  • Kubelet auth: K3s men-disable read-only port 10255; 10250 balas 401 tanpa token. Solusi: SA + ClusterRole dengan resource nodes/metrics (verbs get/list/watch) lalu scrape pakai credentials_file token SA. Verifikasi: kubectl auth can-i get nodes/metrics --as=system:serviceaccount:monitoring:alloy harus yes.
  • Dashboard provisioned kosong total (tanpa panel) bila panel menunjuk datasource template variable yang tidak teresolve. Solusi: hardcode UID datasource hasil provisioning (PBFA97CFB590B2093) di JSON panel.

10. Maintenance

  • Retention: 30 hari, PVC 15Gi (local-path di node tempat pod prometheus dijadwalkan). Pantau: kubectl -n monitoring get pvc prometheus-data.
  • Upgrade image: ubah tag di manifest lalu kubectl apply. Config Alloy divalidasi saat start — crashloop = syntax salah, cek kubectl -n monitoring logs ds/alloy.
  • Tambah app metrics: tambahkan component prometheus.scrape baru (atau discovery.kubernetes role=pod + annotation prometheus.io/scrape) di ConfigMap alloy-config, restart DS.
  • Ubah password Grafana: edit Secret grafana-admin lalu restart deploy/grafana.

11. Ide Pengembangan

  • Expose Grafana via Cilium dedicated ingress + nginx LB + Cloudflare Tunnel (mis. grafana.dosys.my.id) supaya bisa diakses publik dengan auth.
  • Tambah Loki (logging per-pod) — Alloy sudah mendukung loki.source.kubernetes, tinggal tambah pipeline.
  • Alertmanager + alert rules (pod crashloop, node down, disk penuh).
  • Dashboard Hubble/Cilium metrics untuk network visibility antar pod.

No comments to display

Back to top