FRNLEN
Describe your need
Menu
ExpertiseAutomationProjectsPricingAboutBlogContact
FRNLEN
Case StudyInfrastructureSelf-HostingCloud#Kubernetes#k3s#Platform engineering#Cloudflare Tunnel#Kustomize#Backup & recovery

What runs where on the shared Kubernetes platform

A working map of the platform behind this site: four single-node k3s clusters, how requests reach them, how their configuration lives in Git and where the backups go.

Orthrus IT
Map of four single-node k3s clusters: an apps cluster and a tooling cluster for UAT and for production, Cloudflare Tunnel into the apps clusters and private-network links to tooling
Each environment has an apps cluster and a tooling cluster. Requests arrive through Cloudflare Tunnel; metrics, logs and automation webhooks stay on the private network.

This notebook documents the platform that runs this site and a few other web applications. It is written for whoever operates the platform next month, and for anyone wondering whether a small Kubernetes setup makes sense for their own services. It names the components and the reasons behind them. Versions, addresses and hostnames are deliberately left out.

Apps clusters and tooling clusters

There are four clusters. UAT and production each get a pair: one cluster serves the applications, the other runs what is needed to operate them.

  • The apps clusters run the hosted applications. Each application gets its own namespace and, when it stores data, its own PostgreSQL database. This site is a Next.js front end with Payload CMS on PostgreSQL.
  • The tooling clusters run Prometheus, Loki and Grafana for observability, n8n for automation, WAHA so those automations can send WhatsApp messages, and cert-manager for certificates.

We split them for capacity and for incidents. An application that eats all the CPU or memory cannot starve the monitoring that has to explain it, and the tooling side can be restarted or upgraded without touching a pod that serves customers. UAT has the same layout as production, so a release rehearsed in UAT goes through the same routing, storage and automation paths.

One server per cluster

Each cluster is a single k3s server. k3s ships what a small cluster needs in one binary: containerd, CoreDNS, local-path storage and metrics-server. On a single node it keeps the cluster state in an embedded SQLite database instead of etcd.

That means no failover inside a cluster. If a server dies, its workloads stop until we rebuild the cluster from the repository and restore its data. Three control-plane nodes per environment would buy failover the current load does not need, at several times the cost and complexity. We put that effort into the rebuild and restore path instead, and we test it.

How a request gets in

Public traffic reaches the apps clusters through Cloudflare Tunnel. A cloudflared connector runs on each server and keeps an outbound connection open to Cloudflare, which forwards the requests for published hostnames. Inside the cluster, Traefik matches the hostname and path to a Kubernetes Service, and the Service hands the request to a ready pod.

Traffic between clusters stays on the provider's private network. The apps clusters use it to push metrics and logs to tooling. This site uses it to pass each new contact request to n8n, which drafts a follow-up and sends a WhatsApp notification through WAHA.

Everything that runs is declared in Git

The platform manifests live in an infrastructure repository organised with Kustomize: a base per application, then one overlay per environment for whatever differs, such as the image tag, hostnames and resource limits. A few applications keep their manifests next to their code, with the same layout. Secrets are the exception. They are created in the cluster and never committed.

Before any kubectl apply, we compare the overlay with the live cluster using kubectl diff. The repository records the image each environment runs, and a diff that shows more than the expected change stops the deployment until someone understands why.

Backup path from databases, media and cluster state through CronJobs and rclone to an off-site S3 store, followed by a restore test and a twice-daily freshness check
Copies leave the provider every night. A restore test and a freshness check decide whether they can be trusted.

Backups leave the provider

Anything a rebuild cannot recreate is copied to S3-compatible object storage, a MinIO instance outside the cloud provider, over a WireGuard tunnel or HTTPS. Each backup is a Kubernetes CronJob.

  • Every application database gets a nightly pg_dump, kept for 30 days.
  • Files uploaded through the CMS are archived every night and kept for 30 days.
  • The tooling cluster's k3s datastore is copied every six hours, and the last seven copies are kept. The apps clusters are rebuilt from the repository instead.

Uploads go through rclone. Every job image is pinned by digest, and no job downloads a tool when it starts.

We only trust a backup after restoring it. A restore job, run after any change to the backup chain, pulls the latest dump into a throwaway PostgreSQL instance and checks that the expected tables and media entries are there. Twice a day, another job lists every production backup location and sends a WhatsApp message when the newest file is more than 26 hours old.

← All field notes