📦 Infrastructure as Code & Observability

Rebuilt my Docker host from scratch — provisioned by Terraform, configured by Ansible, monitored by Prometheus, Grafana, and Loki. No more clicking through a UI and hoping I remember what I did.

Overview

My lab worked, but nothing about it was reproducible. VMs were created by clicking through Proxmox. Containers were started by hand. If a host died, rebuilding it meant remembering what I did months ago. This phase replaced that with code.

Architecture

Technologies Used

• Terraform (bpg/proxmox provider)
• Ansible
• Prometheus
• Grafana
• Loki + Promtail
• Uptime Kuma
• node-exporter + cAdvisor
• Docker Compose

Highlights

• Provision Proxmox VMs with Terraform, authenticated by a scoped API token rather than root credentials.
• Configure hosts and deploy services with Ansible playbooks — no manual docker run commands.
• Scrape host and per-container metrics every 15 seconds with Prometheus, visualized in Grafana.
• Aggregate logs with Loki, shipped by Promtail reading the Docker socket directly.
• Run independent up/down checks on seven services with Uptime Kuma — including the OPNsense GUI across a VLAN boundary — with email alerting verified end to end.
• Added an external dead man's switch: OPNsense pings healthchecks.io every five minutes, so an outage that takes the lab offline still triggers an alert from outside it.

Challenges

• Writing a playbook to codify the existing setup revealed two containers had been deployed from a forgotten repo, and two others had no config backing them at all. Rather than reverse-engineer it, I backed the VM up, destroyed it, and rebuilt clean — redeploying only the services that were actually configured.
• Notepad silently saved the Terraform .tfvars file in an encoding that broke parsing, with no useful error to point at it.
• Proxmox API tokens need an explicit permission grant even when owned by root — and that grant disappears every time the token is rotated.
• WSL mounts the Windows filesystem with permissions that make Ansible refuse to trust its own config file.
• Docker's {{.Name}} template syntax collides with Ansible's templating engine, which tries to evaluate it first.
• Monitoring the OPNsense GUI meant crossing the VLAN boundary I built in Phase 1. Instead of loosening the rule, I added one scoped exception: a single host, to a single destination, on a single port.
• A real power outage exposed a gap in the monitoring: Uptime Kuma emailed a recovery notice once power returned, but could never have alerted during the outage — it and its mail relay run on the same power and network they were supposed to be watching. I inverted the direction. OPNsense now sends a heartbeat out to healthchecks.io, which alerts when the heartbeats stop, because nothing about its infrastructure depends on mine.
• A misnamed leftover file — terraform.tvfars, a typo of the gitignored terraform.tfvars — sat in the repo's public git history from the first Terraform session, because no .gitignore glob matched the typo. The exposed token turned out to be an already-rotated one, confirmed by searching the full history for the live value, but I treated it as real: rewrote history with git filter-repo and verified on GitHub. The fix was not a better ignore pattern, since no glob catches an arbitrary future typo. It was a gitleaks pre-commit hook that scans file contents for secret-shaped strings regardless of filename.

What I Learned

Destroying a working VM felt wrong until I realized I could not confidently rebuild it — which meant it was already fragile. Infrastructure I cannot recreate from a file is infrastructure I do not really control. The observability work made the same point from the other direction: I had been finding problems by noticing them, which only works while you are looking.

Next Steps

• Build a golden VM image with Packer to remove the last manual step — clicking through the Ubuntu installer.
• Add Grafana alerting on resource thresholds, beyond Uptime Kuma's up/down checks.
• Phase 4: security hardening.

Gallery

✕‹›