🛡️ Keep the server boring

This is the last lesson, and the least exciting one, on purpose. Your server holds, or soon will, things you can't download again: accounts, an email list, a job's results, the keys behind your reminders. One wrong command, a dead disk or a break-in could take them all. I'm a data person, not a server admin, so I want a server too boring to need me at 3 a.m., and a routine for the night it does. It all builds on the server and the infra repo from Deploy behind one front door.

A backup you haven't restored isn't a backup

My server's first backup run reported success. Then I ran the first real restore test: the database dumps, the only files in there I truly couldn't recreate, were missing from the snapshot. No error, no warning.

The cause was one exclude rule, a line telling restic what to skip. My backup folder held the repository, which must never back itself up, and a staging folder where the script leaves the dumps before each backup. I excluded the whole backup folder, and listed staging as a folder to back up, on purpose. No matter: restic applies an exclude to everything under that path, even a folder you named.

Two rules came out of it. Exclude exact paths, never a parent of something you need. And keep a restore-test script beside the backup script, and run it after any change to your backups: "success" only proves the backup ran. Yours, in step 2, restores the newest snapshot, proves the config, secrets and dumps came back identical, and opens every dump. Until it prints PASS, you have a hope, not a backup.

Layer one: every night, restic saves a snapshot to a repository on the same server, keeping 7 daily and 4 weekly. That covers a bad command, a broken deploy, a table emptied by mistake. Layer two: Hostinger's VPS (rented server) plans include a free weekly backup of the whole server, repository included (plans change, so check yours). That covers a dead disk or a lost server.

Before a risky change, like an operating-system upgrade, also take a manual snapshot in hPanel, Hostinger's control panel, or through the Hostinger MCP, Claude's plug-in for your Hostinger account from Your own server. Hostinger keeps one per server, so each replaces the last.

The password comes first, and you make it, not Claude: generate a long one in your password manager, its off-server home from day one, and paste it into your own server tab, where it never appears on screen:

umask 077 && mkdir -p /root/secrets /root/ops
test -e /root/secrets/restic-password || { read -rsp 'Paste the restic password, then press Enter: ' PW && printf '%s' "$PW" > /root/secrets/restic-password; unset PW; }
printf 'RESTIC_REPOSITORY=/root/ops/restic\nRESTIC_PASSWORD_FILE=/root/secrets/restic-password\n' > /root/secrets/backup.env

umask 077 makes new files root-only, like chmod 600; read -s hides your paste; test -e … || never replaces an existing password. The env file (settings, one NAME=value per line) tells restic where the repository and password live.

From here on, work in your infra repo with Claude Code, the AI assistant in VS Code, your code editor (open the folder, run claude in its terminal). Every script lives there first, then gets copied to the server.

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath:

apt update && apt install -y restic sqlite3
set -a && . /root/secrets/backup.env && set +a
restic init

set -a turns each env-file line into an environment variable, a named setting programs read at startup. Now prove your password manager's copy works; this ignores the password file, so restic asks for it:

env -u RESTIC_PASSWORD_FILE restic -r /root/ops/restic snapshots

Paste it from your manager: the right one opens the repository without an error. wrong password or no key found means the copies differ; the repository is still empty, so delete /root/ops/restic and /root/secrets/restic-password and redo this step.

A database writes to its files constantly. Copy one mid-write and the restore can fail with database disk image is malformed, the day you need it. So each database exports itself into staging, and the live files stay out of the snapshot:

• Postgres: pg_dump, run inside its container, writes the database as one file of SQL, the commands that rebuild it.
• SQLite, PocketBase included: the sqlite3 command's .backup makes a clean copy, safe while the app keeps writing.

So loop instead: this script asks every running container whether Postgres answers inside it, then asks Postgres for its databases. One you add next year is in that night's backup.

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath (your file names will differ):

#!/usr/bin/env bash
set -euo pipefail
[ -n "${RESTIC_REPOSITORY:-}" ] || { set -a; . /root/secrets/backup.env; set +a; }
S=/root/ops/staging
rm -rf "$S" && mkdir -p "$S"

# Postgres: every database, in every container where a Postgres server answers
containers=$(docker ps --format '{{.Names}}')
for c in $containers; do
  docker exec "$c" pg_isready -q > /dev/null 2>&1 || continue
  u=$(docker exec "$c" printenv POSTGRES_USER || echo postgres)
  dbs=$(docker exec "$c" psql -U "$u" -d postgres -Atc 'select datname from pg_database where not datistemplate')
  for db in $dbs; do
    docker exec "$c" pg_dump -U "$u" --clean --if-exists "$db" > "$S/${c}__$db.sql"
  done
done

# SQLite: every .db and .sqlite file under /root, copied safely while live
find /root -path /root/ops -prune -o -type f \( -name '*.db' -o -name '*.sqlite' \) -print |
while read -r f; do
  sqlite3 "$f" '.timeout 60000' ".backup $S/$(echo "${f#/root/}" | tr / _).backup" < /dev/null
done

# one encrypted snapshot of /root, then thin out old ones and verify the repository
restic backup /root --exclude-file /root/ops/excludes.txt
restic forget --keep-daily 7 --keep-weekly 4 --prune
restic check

The env file loads only if no repository is set yet, so you can test against a scratch one. set -e stops at the first failure, so a broken dump never ends in a cheerful snapshot. Postgres lists its built-in postgres database too, and .timeout 60000 lets SQLite wait a minute for a busy app.

Write the inventory down

List what can't be recreated. It doubles as your disaster-recovery runbook, the steps you follow on a bad day, so keep it in the repo and on the server:

What this server holds that I can't recreate
- Postgres: every database, dumped to staging/CONTAINER__DB.sql
- SQLite (PocketBase too): every .db/.sqlite under /root, copied to staging/*.backup
- Uploads: /root/pb/pb_data/storage, ordinary files, saved as they are
- Config: /root/caddy/Caddyfile (master copy in infra), compose files, every README
- Secrets: /root/secrets (env files; push keys, without which every reminder subscription dies; the restic password file)
- Skipped: the pipeline's download cache, *.tgz, caches, the live database files
- The restic password is ALSO in my password manager. Without it, nothing restores.

Put something back:
  set -a; . /root/secrets/backup.env; set +a
  restic restore latest --target /tmp/restore-test
- one file: cp -a /tmp/restore-test/root/myapp-data/state.json /root/myapp-data/
- Postgres: docker stop umami; docker exec -i umami-db psql -U umami -d umami < /tmp/restore-test/root/ops/staging/umami-db__umami.sql; docker start umami
- SQLite: docker stop pb; cp /tmp/restore-test/root/ops/staging/pb_pb_data_data.db.backup /root/pb/pb_data/data.db; rm -f /root/pb/pb_data/data.db-wal /root/pb/pb_data/data.db-shm; docker start pb

Replace the names with yours. Each restore stops the app first, and the SQLite one deletes the old -wal and -shm files (unmerged changes that don't belong to the restored copy). Some apps encrypt stored credentials with a key file in their data folder, and a database without its key is useless, so keep those folders whole.

Skip what you can download again: leaving out re-downloadable raw data shrank my server's snapshot from about 3.2 GB to about 510 MB, about 95 MB once compressed.

# the repository itself: an exact path, never a parent of staging
/root/ops/restic
# live database files: backup.sh copies each one safely into staging first
*.db
*.sqlite
*-wal
*-shm
*-journal
# things you can download or rebuild (restic keeps its own cache in .cache)
/root/.cache
*.tgz
/root/myapp-pipeline/cache

A line starting with / is an exact path; the others match that name in any folder, which is why staged copies end in .backup, not .db. Swap the last line for your own re-downloadable folders (this one is the download cache from Big public data on a small server; its raw/ folder of hand-fetched files stays in). Over a few nights, du -sh /root/ops/restic should grow a little; doubling means the repository is copying itself.

Every night, by itself

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath: two small files, copied into /etc/systemd/system:

[Unit]
Description=Nightly backup: dump every database, then a restic snapshot

[Service]
Type=oneshot
ExecStart=/bin/bash /root/ops/backup.sh
Environment=HOME=/root
Nice=19
IOSchedulingClass=idle

[Unit]
Description=Run backup.service every night

[Timer]
OnCalendar=*-*-* 04:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

systemctl daemon-reload && systemctl enable --now backup.timer
systemctl list-timers backup.timer

You should see one row whose NEXT is tonight, in server time, usually UTC, the world's reference clock. Nice=19, the idle disk class and the random delay keep the job out of your apps' way, and Persistent=true runs a missed night once the server is back. Services start without a home folder, so HOME tells restic where its cache lives. The same pattern refreshes the newsletter posts on traqqit.com hourly. No row? The timer was never enabled: rerun the enable --now line.

The finish line: restore it

EXAMPLE — YOU ASK CLAUDE CODE

#!/usr/bin/env bash
set -euo pipefail
[ -n "${RESTIC_REPOSITORY:-}" ] || { set -a; . /root/secrets/backup.env; set +a; }
T=/tmp/restore-test
rm -rf "$T"
restic restore latest --target "$T" > /dev/null

# 1. config, secrets and every dump came back byte for byte (contents never printed)
for d in /root/caddy /root/secrets /root/ops/staging; do
  diff -rq "$d" "$T$d" > /dev/null || { echo "FAIL: $d is missing or different"; exit 1; }
  echo "$d: identical"
done

# 2. every SQLite copy passes SQLite's own check
for f in "$T"/root/ops/staging/*.backup; do
  [ -e "$f" ] || continue
  r=$(sqlite3 "$f" 'pragma integrity_check')
  [ "$r" = ok ] || { echo "FAIL: $f: $r"; exit 1; }
  echo "$(basename "$f"): ok"
done

# 3. every Postgres dump loads into a scratch database, with as many tables as live
q="select count(*) from information_schema.tables where table_schema not in ('pg_catalog','information_schema')"
for f in "$T"/root/ops/staging/*.sql; do
  [ -e "$f" ] || continue
  n=$(basename "$f" .sql); c=${n%%__*}; db=${n#*__}
  u=$(docker exec "$c" printenv POSTGRES_USER || echo postgres)
  docker exec "$c" dropdb -U "$u" --if-exists restore_test
  docker exec "$c" createdb -U "$u" restore_test
  docker exec -i -e PGOPTIONS=--client-min-messages=warning "$c" psql -U "$u" -d restore_test -q -v ON_ERROR_STOP=1 < "$f" > /dev/null
  live=$(docker exec "$c" psql -U "$u" -d "$db" -Atc "$q")
  back=$(docker exec "$c" psql -U "$u" -d restore_test -Atc "$q")
  docker exec "$c" dropdb -U "$u" restore_test
  [ "$live" = "$back" ] || { echo "FAIL: $n: $back tables restored, $live live"; exit 1; }
  echo "$n: loads, same tables as live"
done
rm -rf "$T"
echo PASS

diff -rq compares whole folders and only says whether they differ. Run the test right after a backup (anything changed in between shows as different), and add other folders you can't lose, such as an app's compose files, to the first loop. It needs free disk about one snapshot's size; after a failure, inspect /tmp/restore-test, then delete it. Failing on /root/ops/staging is my bug exactly: an exclude is eating your dumps.

You've had the master-copy habit since Deploy. Now it becomes two scripts, because a habit you must remember fails the day you're in a hurry.

EXAMPLE — YOU ASK CLAUDE CODE

What's happening underneath, in two files you can read in a minute:

#!/usr/bin/env bash
set -uo pipefail
: "${HOST:?first run: export HOST=root@YOUR_VPS_IP}"
cd "$(dirname "$0")/.." || exit 2
T=$(mktemp -d)
ssh -i "${SSH_KEY:-$HOME/.ssh/id_ed25519}" "$HOST" 'cat /root/caddy/Caddyfile' > "$T/live" || { echo "FETCH FAILED"; exit 2; }
git -c core.autocrlf=false diff --no-index --ignore-space-at-eol "$T/live" Caddyfile | sed -E 's/\$2[aby]\$[^[:space:]]+/HASH_HIDDEN/g' | tee "$T/diff"
if grep -v '^---' "$T/diff" | grep -q '^-'; then
  echo "DRIFT: the - lines exist only on the server. Put them in the repo first."
  [ "${FORCE:-0}" = 1 ] && echo "FORCE=1: continuing anyway." || exit 1
else
  echo "OK: nothing on the server is missing from the repo."
fi

#!/usr/bin/env bash
set -euo pipefail
: "${HOST:?first run: export HOST=root@YOUR_VPS_IP}"
KEY=${SSH_KEY:-$HOME/.ssh/id_ed25519}
cd "$(dirname "$0")/.."
bash scripts/caddy-check-drift.sh
scp -i "$KEY" Caddyfile "$HOST:/root/caddy/Caddyfile.new"
ssh -i "$KEY" "$HOST" 'docker run --rm -v /root/caddy:/etc/caddy:ro caddy:2 caddy validate --config /etc/caddy/Caddyfile.new --adapter caddyfile &&
  mkdir -p /root/backups && cp /root/caddy/Caddyfile /root/backups/Caddyfile.$(date +%F-%H%M) &&
  mv /root/caddy/Caddyfile.new /root/caddy/Caddyfile &&
  docker exec -w /etc/caddy caddy caddy reload'
git diff --quiet HEAD -- Caddyfile || git commit -m "Apply Caddyfile" -- Caddyfile

Run them from Git Bash in infra (Mac or Linux: any terminal) after export HOST=root@YOUR_VPS_IP, with your server's address, once per terminal; set SSH_KEY too if your key isn't the default. Settings from the environment let the scripts run from any machine.

In the diff, a + line is your change, waiting to go live. A - line exists only on the server, so the check stops: copy it into the repo, commit, rerun. Deleted a block on purpose? Read the - lines, then run FORCE=1 bash scripts/caddy-apply.sh. From now on, every front-door change is an edit plus bash scripts/caddy-apply.sh, and any other deploy that rewrites the server's copy runs the drift check first. The sed keeps password hashes (one-way scrambled passwords; step 5 adds one) out of chat logs, but keep the repo private, with *.env in its .gitignore: real secrets never go in git.

One root-only env file per service, in /root/secrets from Deploy (chmod 600), handed over with --env-file (a container) or EnvironmentFile= (a timer's service) and read as environment variables (process.env.NAME in Node). It stays outside every folder you copy, serve or commit, and its values never go into a terminal or an AI chat, where they'd stay for good. Four moves cover nearly everything:

umask 077
echo "COOKIE_SECRET=$(openssl rand -hex 32)" >> /root/secrets/myapp.env
grep '^SMTP_PASS=' /root/secrets/other.env >> /root/secrets/myapp.env
sed 's/=.*/=***/' /root/secrets/myapp.env
ls -l /root/secrets

Line 2 writes a random secret straight into the file; line 3 copies a line from another env file unseen (use your own names); line 4 masks the values; the listing should show -rw------- and root for every file. If Claude refuses to print a password, that's its safety check working: have it edit the file in place (sed -i).

Read-only keys, and rotation in four moves

Where your app only reads from a service, use a read-only key, so a leak can't spend money or place orders; one of my private dashboards only reads, so its key is now read-only. To rotate a key (replace it so a leaked copy stops working):

1. Create the new key at the provider.
2. Put it in the env file, recreate the container with the command in its README, and check a live page that uses it.
3. Revoke the old key at the provider.
4. Delete any backup copy of it. Can't revoke today? Set a reminder: old keys forgotten in backup folders leak months later.

Jobs don't sign in as the admin

PocketBase emails a "login from a new location" alert when an account signs in from an address and browser it hasn't seen. A scheduled job (a cron job) signing in as the superuser, PocketBase's all-powerful admin, from a fresh container can look new every run, and a stream of alerts trains you to ignore the real one. Give each job its own account with only the access it needs, or at least switch the alert off for superusers only (a setting on _superusers, PocketBase's table of admins; ask Claude where your version keeps it).

EXAMPLE — YOU ASK CLAUDE CODE

Rebuild the timeline from the logs before you change anything: restarting first wipes evidence, and a hunch often fixes the wrong problem. Four places to read:

journalctl -k --since '2 days ago' | grep -i -E 'out of memory|oom-kill'
journalctl -u myapp-ingest --since yesterday
journalctl -u docker --since '02:00' --until '03:00'
docker logs --timestamps --since 12h myapp-api

In order: memory kills (-k reads only since the last boot; after a reboot, add -b -1); one service's starts, stops and errors (your names, like backup); Docker's own log around the event; one container's output, with times. Then look for gaps: an hourly job with no 01:00 or 02:00 line stalled there. Some lines carry two times, the journal's and the program's time=; a big gap between them means the machine froze. Login alerts from an internal address can be your own delayed jobs reconnecting after a stall, so match them against those logs before you rotate every password.

EXAMPLE — YOU ASK CLAUDE CODE

Lock private pages, and only those

Reports and admin pages don't belong on the open web, and an obscure subdomain won't hide them (Launch, honestly showed how anyone can list your padlocked names). Caddy's basic_auth adds a real lock (spelled that way since Caddy 2.8; old tutorials say basicauth, and docker exec caddy caddy version shows yours).

Make the hash in your own server tab: it asks for the password twice, shows nothing, and prints a line starting $2a$. The password goes in your password manager:

docker exec -it caddy caddy hash-password

Add these lines inside your app's block in infra/Caddyfile, at its top level rather than inside a handle (a group of lines for some paths; Caddy runs basic_auth before those), replacing both placeholders:

@locked path /internal /internal/*
basic_auth @locked {
	YOUR_USERNAME YOUR_BCRYPT_HASH
}

@locked is a matcher, a named list of paths; only those ask for a password. Ship it with bash scripts/caddy-apply.sh, then prove both sides with your username and domain (in PowerShell, type curl.exe):

curl -sI https://myapp.yourdomain.com/internal/
curl -u YOUR_USERNAME -sI https://myapp.yourdomain.com/internal/

The first answers HTTP/2 401; the second asks for the password, then answers HTTP/2 200 (or 404 while /internal has no index.html: you're past the lock either way). Add every new private path to @locked, or it's public. Lock only what needs it: on a private, business-facing version of one of my tools, the analyst pages are locked but the page people need in a hurry is open on purpose, because a password there is exactly the friction the tool exists to remove. Private pages also get <meta name="robots" content="noindex">, which asks search engines to skip them.

The README you'll need at 3 a.m.

Plain docker run is fine for a side service if a README (a plain notes file) beside it records the exact command, where its env file lives (never its contents) and its front-door block, as in Your own server; my analytics service runs that way. (docker inspect NAME shows a running container's settings, secrets included, so filter those out before Claude reads it.) Put both routines where half-asleep you will find them:

## Something broke: change nothing until the timeline is written
1. journalctl -k --since '2 days ago' | grep -i -E 'out of memory|oom-kill'   (rebooted? add -b -1)
2. journalctl -u SERVICE --since yesterday      (every service involved)
3. journalctl -u docker --since HH:MM --until HH:MM; docker logs --timestamps --since 12h NAME
4. Gaps in each job's own log. Login alerts from inside? Match them to these first.
5. Write the timeline here, oldest first. Then fix the cause.

## Rotate a key
New key in the env file, recreate the container, check it live, revoke the old key, delete its backup.

Admin without a back office

Admin panels grow forever: users, then payments, then refunds. Mine is one static page that signs in as the PocketBase superuser and calls a few superuser-only routes (addresses your server code answers): headline counts, one user looked up by email. Everything else is a link out, to the Stripe Dashboard (payments, from Get paid without leaking money) and PocketBase's own dashboard. One route, as a PocketBase hook (your own code running inside PocketBase):

routerAdd("GET", "/api/myapp/stats", (e) => {
  const live = ($os.getenv("STRIPE_SECRET_KEY") || "").startsWith("sk_live_")
  return e.json(200, { users: $app.countRecords("users"), stripe: live ? "live" : "test" })
}, $apis.requireSuperuserAuth())

The last line is the lock: a request (one call to the server) without a superuser token, the proof of login, gets 401. The key's prefix picks Stripe's test or live dashboard links, so the page switches itself the day you go live, and the key never leaves the server. The page sits in /internal behind basic_auth, with its token in sessionStorage, browser storage that empties when the tab closes.

EXAMPLE — YOU ASK CLAUDE CODE

Phone-friendly, no chart library

You'll check this on your phone, where wide tables and fixed-size charts break first. A sparkline (a tiny line chart without axes) is one line of SVG, the web's drawing format, and its viewBox lets it shrink:

<svg viewBox="0 0 100 24" preserveAspectRatio="none" width="100%" height="24">
  <polyline points="0,20 25,14 50,16 75,6 100,9" fill="none" stroke="currentColor"/>
</svg>

@media (max-width: 600px) {
  th:nth-child(4), td:nth-child(4) { display: none }
  .tiles { grid-template-columns: 1fr 1fr }
}

Never signal good or bad by color alone: write ▲ 12%, ▼ 8% or words. Check at 390px in DevTools' device mode (the browser's developer tools): F12, then Ctrl+Shift+M in Chrome or Edge; Mac: Cmd+Option+I, then Cmd+Shift+M; Safari: Develop menu, Responsive Design Mode.

My admin page also has a live visitor globe. Umami, the analytics tool, stores city names, not coordinates, so a one-off script built a lookup table of about 65 KB from free data (country centre points, plus large cities from GeoNames, a free place-name database), and the tiny cobe library draws it.

Proxy tricks that save a rebuild

• A link inside an app you can't edit. My analytics dashboard is a prebuilt image, so a tiny nginx container (another web server) in front of it adds my link before </body> with sub_filter, its find-and-replace. Only dashboard pages pass through it; the tracking script and API keep the direct route, so data collection can't break. nginx can't edit compressed pages, so it clears the Accept-Encoding header, the line in a request that offers to accept compressed answers.
• An icon without a rebuild. The price tool and TravelMap both showed a generic favicon (the tab's little icon), and both are slow to rebuild. Browsers ask for /favicon.ico by themselves when a page names no icon, so Caddy answers that path with an SVG: one reload, no rebuild. Deploy's myapi block, with the icon:

myapi.yourdomain.com {
	handle /favicon.ico {
		header Content-Type image/svg+xml
		respond `<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><circle cx='16' cy='16' r='14' fill='teal'/></svg>` 200
	}
	handle {
		reverse_proxy myapi:8000
	}
}

Backticks keep the SVG's quotes intact, and the final handle sends everything else to the app; check in a private window, since browsers cache icons hard. Last trick: a second brand on the same code gets its own colors by redefining a few CSS variables (named values like --brand that the stylesheet reuses) in one block the server adds only for that brand, not via a class on the page's <body>: on one of my tools, 5 of the second brand's 11 pages had no such class and would have kept the wrong colors.

Do it with Claude: the prompts

The backbone of this lesson is three prompts. Check each one the way its step shows:

Write backup.sh: pg_dump every database in a loop, sqlite3 .backup for PocketBase, then restic backup excluding the repository itself. And restore-test.sh that restores the latest snapshot to /tmp and diffs it.

Add a systemd timer for it with Persistent=true, Nice=19 and idle disk priority.

Diff the server's Caddyfile against the repo copy and stop if the server has anything we don't.

What's happening underneath: restic splits files into chunks, stores each distinct chunk once, compresses it and encrypts it. Each night adds only what changed, so nightly snapshots of mostly unchanged files stay small. Push back when Claude's version excludes a parent of staging, names databases one by one, copies live .db files, prints a secret to "check" it, or calls a backup good without a restore.

Your turn

• Tomorrow, journalctl -u backup -n 20 shows last night's timer run ending in no errors were found, and systemctl list-timers backup.timer shows tonight's.
• bash /root/ops/backup.sh && bash /root/ops/restore-test.sh prints PASS. Then replay my bug: add the line /root/ops to /root/ops/excludes.txt and rerun both. Done when it fails on /root/ops/staging, then passes once you delete that line.
• env -u RESTIC_PASSWORD_FILE restic -r /root/ops/restic snapshots opens with the password from your manager, and Hostinger lists a weekly backup of your server.
• bash scripts/caddy-check-drift.sh prints OK. After ssh root@YOUR_VPS_IP "echo '# drift test' >> /root/caddy/Caddyfile" (from Git Bash, with your address), done when it stops with DRIFT on -# drift test, then prints OK after FORCE=1 bash scripts/caddy-apply.sh.
• ls -l /root/secrets shows only -rw------- root files, infra/NOTES.md holds both routines, and a private page answers 401 without the password, 200 with it.

That's the course. Your server is boring now, which was the point: the interesting part is the next app. Pick it with Find an idea the data can carry; the front door, the backups and the drift check are already waiting, and the loop will catch its database on the first night.