Monitoring runs on the tooling clusters, away from the applications it watches. When an application misbehaves, the tools needed to explain it keep running on another server, with their own CPU and memory.
How metrics reach the central Prometheus
Each apps cluster runs a small Prometheus that scrapes the kubelet and cAdvisor every 30 seconds, along with any pod or Service that opts in through annotations. Every series gets a cluster label and an environment label, then goes through remote write to the Prometheus on the tooling cluster, over the private network.
The local Prometheus is little more than a forwarder with a write-ahead log. A short network outage delays data instead of losing it.
The central Prometheus keeps 30 days, capped at 5 GB. It also runs sources of its own: kube-state-metrics, which turns Kubernetes objects into metrics (replicas, restarts, CronJob results), and node-exporter on the tooling servers.
Logs: Promtail to Loki
Promtail runs on every server as a DaemonSet. It reads container logs, labelled with namespace, pod and container, plus the systemd journal, and pushes everything to Loki on the tooling cluster. Production keeps two weeks of logs.
Loki indexes labels rather than the log text, which keeps it light to run. A query narrows the stream first, then filters lines. {namespace="shop-prod", container="api"} |= "timeout" returns the API lines that mention a timeout, and wrapping the same selector in count_over_time turns them into a graph.
Dashboards are files in the repository
Grafana loads its data sources and dashboards from provisioning files kept in the infrastructure repository, so a rebuilt Grafana comes back with the same dashboards. Three of them answer most questions today:
- Cluster Overview shows pods that are not running, container restarts, CPU and memory per node, and pods per namespace.
- Traefik Overview shows requests per second, the share of 5xx responses and 95th-percentile latency for each service.
- An API dashboard for one of the hosted applications shows request rate, errors and latency per route, plus Node.js memory and event-loop lag.
An alert that watches backups
A dashboard only helps when someone looks at it. Alerts on every CPU spike would mostly be noise, so alerts are reserved for failures that stay hidden until the worst moment. A backup that stops without anyone noticing is the clearest example.
Twice a day, a job lists every production backup location. If the newest file in one of them is more than 26 hours old, it sends a WhatsApp message through WAHA, on the channel that also announces new contact requests. Every Monday morning it also sends a short all-clear, so a quiet week means the checker itself has stopped.
The same data checks a release
After a rollout we look at three things: whether the Traefik error rate moved, what the new pod logged in its first minutes, and whether anything restarted. The review takes a few minutes and belongs to the UAT checklist described in the release entry.