Since 2016 I’ve run my own LAMP server for personal and freelance projects. It hosts this site, my recipe manager, a custom audiobook player and a client’s exam-management app. It started life on a Raspberry Pi and now lives on a Mac mini running Debian.
It’s also my lab. Anything I want to learn properly (Kubernetes, real monitoring, backups I actually trust), I learn here first, on something with real users.
Monitoring started as a Docker Compose stack: Prometheus, Grafana, node_exporter and mysqld_exporter, with named volumes and automatic restarts. When I wanted to learn Kubernetes, I moved that stack over in phases, with zero downtime.
I installed a lightweight k3s cluster right next to the existing Compose stack, and switched off its bundled Traefik ingress so it wouldn’t fight Apache for ports. Each service became proper Kubernetes objects (Deployments, a DaemonSet for node_exporter, persistent volume claims), while the Compose stack kept running as a live fallback the whole time.
Grafana’s database (dashboards, alert rules, contact points) moved over with a careful stop, copy and restart, losing nothing. Then Apache’s reverse proxy cut over to the new Grafana, and once everything checked out end to end, the old Compose stack was retired.
Backups are encrypted with restic and shipped over SFTP to a second physical machine. The SSH key they use is restricted to SFTP only, so even if it leaked it couldn’t be used for anything else.
A backup job that finishes isn’t the same as a backup that works. So every month, an automated job restores the database dumps into throwaway databases and checks file counts against the live data, catching real corruption instead of just a happy exit code. systemd timers run the whole routine (nightly backups, weekly integrity checks, monthly restore tests) with tiered daily, weekly and monthly retention so storage doesn’t grow forever.
What gets backed up lives in one short config file that every script reads, so adding a new project to the regimen is a single edit.
When I extended Grafana alerting to cover backup health, it immediately found two problems nobody knew about: a broken SMTP credential that was silently blocking alert emails, and a stale network setting that had quietly stopped database monitoring for about two days.
Every alert gets tested end to end against real data, safely: temporarily lowering thresholds, or pinning a CPU-heavy process to one core with taskset to trigger a load alert on purpose. One false alarm turned out to be PromQL’s filter semantics tangling with “no data” handling, so I fixed the query instead of muting the alert.
Grafana is reachable from outside, but only through an Apache reverse proxy with Let’s Encrypt TLS and HTTP Basic Auth in front of Grafana’s own login.
MariaDB only listens to the Docker subnet and the Kubernetes pod subnet, each with its own firewall rule and its own grant. The monitoring user can only read instrumentation tables, and passwords live in locked-down environment files, never in version control.