I run two layers of home infrastructure as code: a MikroTik network that
carries the traffic, and a containerised service stack that sits on top of
it. Both rebuild from an empty device. Most of what follows is the parts
that went wrong — seven failures, how each was diagnosed, and the decision
each one forced.
Split on purpose — different blast radius, different lifecycle.
The system
The two halves are separate repositories, and that is a decision rather than an
accident. A firewall change can take the whole house offline; a container image
bump cannot. Different blast radius, different rehearsal process, different
release cadence — so they do not share a repository or a run.
Network layer
ansible-network-template
Two MikroTik devices configured entirely over the RouterOS API — no SSH
transport for configuration. A tagged-VLAN PPPoE uplink, a VLAN-filtered
bridge split into main and servers, a mirrored VLAN
table on the switch, DHCP and DNS for both segments, a WireGuard
road-warrior server, and a firewall that intercepts plaintext DNS, blocks
DoT/DoH bypass and enforces the segment boundary.
6 roles · 10 ADRs · rehearsed on a disposable RouterOS CHR VM before hardware
Eight Ansible roles deploying a containerised stack onto an AlmaLinux 9 VM:
an ad-blocking DNS resolver, a Caddy reverse proxy built with
xcaddy for wildcard certificates over ACME DNS-01, a password
manager, a service dashboard, a Prometheus and Grafana stack with dashboards
provisioned as code, this website, and a Cloudflare Tunnel that exposes the
domain with no inbound port open anywhere.
8 roles · 10 ADRs · rehearsed on a Vagrant VM before production
Each of these was diagnosed from the symptom rather than looked up. Where a
failure changed how the code works, the decision record is linked next to it.
Network layer
A clean test run that proved nothing
The instant vlan-filtering was enabled on the RB5009, every
access port kept link and lost all layer-3 connectivity. A VLAN-filtering
bridge treats itself as a port — the CPU port — and the gateway address sits
directly on the bridge, untagged. Listing the bridge as tagged
for that VLAN tells it to expect 802.1Q frames that never arrive. The rule I
extracted: bridge membership follows where that VLAN's address is reached
from, not a blanket assumption that the CPU needs every VLAN.
The part worth keeping is why the rehearsal missed it. The test VM's
management path sits outside the bridge on purpose — that is what makes a
wedged VM recoverable. Making the test able to catch this bug would cost the
rehearsal its safety net. So the check moved to a hardware review against
three explicit cases instead of pretending the green run meant something.
Updating the switch's default gateway added a route rather than modifying the
existing one, leaving two. RouterOS exposes ip route and
ip firewall mangle without a primary key, so the module has
nothing to match an existing entry against. The obvious fix — letting Ansible
own the whole path and delete anything unlisted — would have removed entries
the role has no business touching. Instead each managed entry carries an
ansible:<purpose> comment and a paired cleanup task drops
the stale one after the replacement is in place.
The CRS310 could not be reached from the router for its first configuration
run. VLAN 1 is tagged on the trunk between the two devices, and a
factory-default switch speaks untagged frames only — so the link was up and
the device was silent. It has to be bootstrapped with a cable straight to a
laptop and moved into place afterwards. Written into the bootstrap
documentation rather than remembered, because a fresh deployment hits it
again.
The Grafana network panel showed plausible numbers that did not match
reality. node_exporter running without
network_mode: host sees only the interfaces in its own container
namespace, and mounting the host's /proc does not fix it —
/proc/net is per-namespace, not per-mount. Diagnosed by
comparing the exported metrics against /proc/net/dev on the host
directly. Rather than leave a panel that looked right and was not, the panel
was removed; CPU, memory and disk metrics are accurate and stayed.
Rebuilding an image under an unchanged tag leaves the running container
pointing at a digest that no longer exists, and Compose fails with
No such image at the next refresh. Separately, a single file
bind-mounted into a container binds to one inode at start — and editors that
save by writing a replacement, which includes nano and
sed, produce a new inode, so the container carries on reading
the orphaned original while the disk already shows the new content.
Both present as configuration problems and neither is. Both need the
container recreated, not restarted — a distinction that is invisible until it
costs you an hour.
cAdvisor started cleanly, reported healthy and saw no containers at all.
Versions below v0.54.0 do not understand how the containerd snapshotter
stores container layers. The upgrade had its own trap: the image registry
moved from gcr.io to ghcr.io at v0.53.0, so bumping
the tag without changing the registry fails the pull instead. Version and
registry are now pinned together, with a comment in the role explaining why
neither may be changed alone.
Reviewing this site turned up that www.kaletkadev.com had no DNS
record at all. I created it before the tunnel had a matching ingress rule —
the wrong order. Negative caching hid the mistake for thirty minutes, after
which the name began resolving and returning 404 until the configuration
shipped. A self-inflicted outage window, small and entirely avoidable: the
backend accepts traffic first, then you point traffic at it.
Twenty decision records across the two repositories. Each one names the option
taken, the options rejected, and the consequences — including the unwelcome
ones. Six of the more interesting are below.
Selected architecture decision records with their trade-offs
Availability now depends on a third party's edge, and all traffic is visible to them in transit. Accepted to keep a credentials store off the public Internet.
A strict policy applied site-wide would break Grafana and the password manager. The narrower scope means each new public service needs its own decision.
A change is not finished when it works. The third consecutive Ansible run has
to report changed=0, or the role is not idempotent and the
change does not merge. Every pull request runs ansible-lint, yamllint and a
playbook syntax check. Roles are rehearsed on a disposable VM — a RouterOS
CHR instance for the network, Vagrant for the services — before anything
reaches hardware.
Working on a router with no console
Neither MikroTik device has a serial console, so a bad firewall or VLAN
change means a factory reset and a re-cable. Three layers guard every such
run: a scheduled rollback armed before the change and disarmed only after
connectivity is confirmed, a RouterOS Safe Mode session held open alongside
the run, and the management port deliberately kept out of the bridge until
the new path is proven. Two lockouts in one day is the signal to stop for the
day rather than push through.
Secrets
Vault-encrypted variables, .example templates as the only
committed configuration, and 0600 on any rendered file that
contains a credential. That last rule exists because a file was once written
world-readable — it is enforced now, and written into the repository's
context file so it stays enforced.
Where AI fits, and where it does not
Both repositories carry a CLAUDE.md context file that constrains
AI-assisted work: never commit real secrets, never weaken a lint rule to make
a hook pass, respect the role ordering that deployment depends on, and a
record of the traps this codebase has already fallen into. Generated changes
go through the same review, the same CI and the same rehearsal VM as anything
else. The guardrails are the interesting part, not the generation.
A GPU node I switched off
I built a headless node around an RTX 5070 attached over OCuLink and ran
local language models on it to see what the hardware could actually do. It is
retired. The node sat idle nearly all the time, and idle hardware does not
justify its power draw or the maintenance surface it adds. Knowing when to
switch something off is the part I would rather be judged on.
This page is part of the system
What you are reading is served by the Caddy instance described above, from the
same AlmaLinux VM, through the same tunnel — no inbound port open anywhere. It
is a separate public repository, deployed by the same playbook run as everything
else.
It also gets treated like production. A review from a senior engineer turned up a
missing www record and no security headers; both are fixed, both in
Git, both with a decision record explaining the trade-offs — why HSTS is not on
the browser preload list, and why the content policy is scoped to this page
rather than the whole domain.