Overview

It re-exposes the primitives every managed platform automates: a process running on a machine, a reverse proxy, TLS, a firewall, a restart policy, and reproducible config. The managed platform is an abstraction over exactly those things, and self-hosting one service shows what that abstraction costs to peel back (and why it sometimes leaks). Example 41 stands the same app up both ways to make the trade-off concrete.

Because every later step -- SSH, the DNS A record (Example 13), the ACME TLS order (Example 14), the PaaS deploy (Example 26) -- needs that one stable value. Recording it (Example 1's .box-ip) is what makes the setup reproducible: a box whose address lives only in a browser tab cannot be rebuilt from memory, which is why Example 42's disaster rebuild depends on the IP (and the setup) being captured.

A key pair is unguessable and scoped to one machine; a password is brute-forceable and reuseable across boxes. Disabling password login closes the single most common remote-compromise vector -- the bots that hammer port 22 trying passwords (Example 43's fail2ban then deals with the residual log noise from those still-trying bots, which can now never succeed).

A non-root user (Example 3), key-only SSH (Example 2), and a firewall (Example 4) -- plus, later, proxy security headers (Example 22) and fail2ban (Example 43). Defense in depth matters because each layer blocks a different attack class, so a failure in one (a forgotten open port, a leaked key) does not hand the attacker the whole box; the other layers still stand.

Default-deny (ufw default deny incoming, Example 4) blocks every port except the few explicitly allowed (22/80/443). Because nothing is reachable unless you said so, an "open port" becomes a deliberate, listed rule -- which Example 23 then audits down to the true minimum, catching any service that silently bound a new port.

To separate "does the app even work" from "does the supervision work." If the public endpoint later fails, you already know the app answers curl locally, so you can narrow the bug to the layer you just added (the unit, the proxy, or TLS) instead of guessing across all of them at once.

Reproducible configuration: the unit declares the start command, the user, the restart policy, and the boot hook, turning "a process I launched by hand" into "a service the OS owns." That is what makes restart-on-crash (Example 10), boot-resilience (Example 36), resource limits (Example 38), and the watchdog (Example 68) expressible at all -- none of which a one-off shell command gives you.

systemctl status and journalctl -u (Example 9). The journal is the single, unified log stream systemd captures for EVERY unit's stdout+stderr, so it is one place to look for every service -- rather than each app inventing its own log location, which you then have to hunt for per service.

The proxy centralizes TLS, virtual hosting (multiple services on one :443, Example 37), security headers (Example 22), rate limiting (Example 56), and graceful drain (Example 74) at one chokepoint. Binding the app to 443 means re-implementing all of that inside the app; the proxy keeps the app simple and lets one box serve many services behind one public face.

It replaces manual openssl cert generation and the renewal-cron that followed -- Caddy obtains, serves, and renews a Let's Encrypt cert automatically on first request (Example 14). That automation is the difference: a 90-day cert that renews itself needs zero ongoing effort, so HTTPS becomes a one-line config change instead of a recurring manual task.

The A record maps the name to the box's IP (Example 13), which is what makes the URL human-usable -- and ACME's HTTP-01 challenge proves domain CONTROL by serving a token from that name, so without the DNS record no certificate can be issued. DNS is the prerequisite for both the name and the cert.

It lets the IDENTICAL artifact run in staging and prod -- the difference between two environments becomes a set of environment variables (Example 15's app.env), not a second copy of the application, so you deploy the same image everywhere and vary only its config (Example 29 does this on a PaaS, Example 61 templates it).

Git history is forever -- even a deleted-and-force-pushed commit may already be cloned or cached, so "it was only one commit" is no defense. The discipline (Example 16) is to keep every secret in a mode-0600 file the repo never sees (/opt/myapp/secrets.env), and to add a scan gate (Example 64's pre-commit hook) that blocks the slip at the keyboard before it lands.

It closes the gap between "it crashed" and "someone told us it crashed" -- a probe hits /health every minute and logs a failure. It does NOT close the "is it degrading before it fails" gap, which is what metrics (Example 39) and alerting on a threshold (Example 67) add. Logs + a health check are the floor; metrics are the next step up.

A CRASH (the process exits) is caught by Restart=always (Example 10); a HANG (the process is alive but stuck) is caught by the systemd watchdog (Example 68), which force-restarts a service that stops pinging within WatchdogSec. A reboot is the third case, handled by enable + WantedBy (Example 8, drilled in Example 36). All three make "alive" the default state without a human.

The build (it compiles your image from a buildpack or Dockerfile, Examples 27-28), the release (it runs the new image under its own supervisor), the proxy + TLS (it attaches a domain and obtains a cert, Example 30), and the rollback history (Example 31). One push does the work of setup + unit + Caddyfile + ACME -- the exact trade-off Example 41 names.

A buildpack (Example 27) detects your language and supplies the runtime -- zero image authoring, but constrained to what it detects. A Dockerfile (Example 28) is fully explicit and portable to any container runtime, but you own every layer. Both are reproducible build recipes; the difference is convenience (buildpack) versus control and portability (Dockerfile).

A naive stop kills the process (and its in-flight requests) the instant the new one is not yet serving. Graceful shutdown (Example 50/74) drains in-flight work before the process exits, and a blue-green swap (Example 73) keeps the OLD instance answering until the NEW one is proven healthy -- so no request ever hits a stopped backend.

Because a backup that has never been restored might be torn, truncated, or empty, and you only find out under duress. The two habits: integrity-check the backup AT BACKUP TIME (Example 34) and run a PERIODIC restore drill (Example 35, automated in Example 77) -- so a corrupt backup is caught when the stakes are zero, not during the first real data loss.

Self-hosting buys control and cost savings at the price of operational responsibility (patching, backups, 3am uptime); a managed platform absorbs that responsibility at the price of control and a higher per-app spend. It is a trade-off because no side is strictly better -- the right answer depends on whether you value the control/cost (self-host) or the absorbed ops burden (managed) for THIS workload, as Example 41 and Example 45 spell out.

It turns "the box works" into "the box works AGAIN, on fresh hardware, without me." A box whose setup is tribal knowledge is unrecoverable in any practical sense; a box whose setup is a script (or a cloud-init user-data, Example 78) is rebuildable -- which is the entire premise of Example 42's disaster-rebuild drill and the capstone's acceptance bar.

A stateful, high-availability, or compliance-bound workload is a poor first self-host -- a managed platform absorbs the backups/failover (stateful+HA) or already carries the compliance cert (compliance) you would otherwise own at 3am (Example 45). The discipline is to name the deciding force (state, HA, compliance, team ops-appetite) rather than default to self-hosting everything.

The unit is missing Restart=always (co-15; Example 10, kata 1). Without it, the default Restart=no applies and a crash stays dead until a human restarts it. Add Restart=always (with a StartLimitBurst cap) so the box heals itself.

The script enabled the firewall BEFORE allowing port 22 (co-04/co-05; kata 2). The default-deny policy went live with SSH still closed, dropping your session. The fix is order: ufw allow 22/tcp FIRST, then ufw --force enable. Recovery requires console/access from the provider, since SSH is now locked out.

The proxy and the app disagree on the port (co-09; Example 12, kata 3). The proxy is forwarding to a local port the app is NOT bound on (e.g. 8001 instead of 8000). Fix the reverse_proxy upstream to match the app's actual bind port.

This is a co-13 hard-iron violation, and it is permanent (history). The secret must be ROTATED (not just deleted), the repo cleaned, and a pre-commit hook (Example 64) added to block the next slip. Going forward, secrets live only in a mode-0600, gitignored file (Example 16), never in the repo.

The unit is not enabled (co-15; Example 8). start runs it now; only enable (the WantedBy symlink) makes it start on BOOT. Run systemctl enable myapp (and systemctl enable caddy -- the proxy must also survive the reboot, or the app is unreachable, Example 36).

The HTTP-to-HTTPS redirect is missing (co-10; Example 53). Without it, a typo'd http:// URL serves content unencrypted, enabling a silent downgrade. Add the redirect block (and HSTS, Example 54, to make browsers stop trying HTTP at all).

The script is not idempotent (co-21; Example 19 vs Example 20). A second run should be a no-op where its postconditions already hold. Wrap each step in an ensure_* guard (Example 20) so re-running changes nothing observable.

Missing log rotation (co-14; Example 24). A logrotate rule (rotate, keep N, compress) bounds the logs, and a SystemMaxUse cap bounds the journal. A full disk presents as a dozen unrelated failures whose single root cause is "no space."

A PaaS rollback (co-16/co-18; Example 31). The PaaS keeps a release history, so one command restores the last known-good image. (On the self-hosted side, the equivalent is a blue-green flip back to BLUE, Example 73.)

An untested backup is a wish (co-19; Examples 34-35, kata-less but central). Add an integrity check at backup time and run a periodic restore drill (automated in Example 77) -- so a corrupt backup is caught now, not during the first real data loss.

The process is hung, not crashed (co-15; Example 68). Restart=always only catches an EXIT; a hung process stays "active" forever while serving nothing. Add a systemd watchdog (WatchdogSec) so the service must ping periodically or be force-restarted.

This is the wrong workload to self-host (co-22; Example 45). A stateful, must-not-lose, HA-adjacent database is a poor first self-host -- a managed Postgres absorbs the backups and failover you would otherwise own at 3am. Name the deciding force (stateful + HA), not the team's newfound skill.

[Service]
Restart=always
RestartSec=3
StartLimitIntervalSec=60
StartLimitBurst=5

Root cause: the unit relies on systemd's default Restart=no (co-15; Example 10, kata 1). Adding Restart=always makes a crash self-healing; StartLimitBurst caps it so a hopeless crash loop trips into failed instead of burning the box. The "after" fixture adds exactly these lines.

ufw allow 22/tcp        # allow SSH FIRST
ufw allow 80/tcp
ufw allow 443/tcp
ufw --force enable      # NOW enabling is safe

Root cause: order is load-bearing. The default-deny policy is LIVE the moment enable runs, so SSH must already be allowed or your session is one of the things denied. Recovery requires console/access from the provider, which is why the bug is so costly (co-04).

myapp.example.com {
  reverse_proxy 127.0.0.1:8000   # match the app's actual bind port
}

Root cause: the proxy and the app disagree on where to meet (co-09; Example 12). A 502 with both sides healthy is almost always a port/path mismatch between reverse_proxy and the app's bind address. Verifying the app directly (curl 127.0.0.1:8000/health) isolates it from the proxy first.

APP_HOST=127.0.0.1
APP_PORT=8000
# No secret here. The real APP_SIGNING_SECRET lives ONLY in:
#   /opt/myapp/secrets.env   (mode 0600, gitignored), read via EnvironmentFile=-

Root cause: this violates the hard-iron rule (co-13; Example 16). The fix is two-sided: move the secret to a mode-0600, gitignored file set out of band, AND rotate the leaked value (deleting it from one commit does not undo history). A pre-commit hook (Example 64) blocks the next slip at the keyboard.