Skip to content

Backup system overview

Everything funnels to the NAS (192.168.1.5), and the VM dumps get a second copy in Google Drive. A small always-on container called ops (192.168.1.190, on the Proxmox OptiPlex) runs all the machinery: the Kopia repository server, the nightly off-site sync, monitoring, and (gated-off) wake-on-LAN. Built August 2026.

flowchart LR
    subgraph src["Sources"]
        mbp["MacBook<br/>Docs · Desktop · Pictures<br/>hourly"]
        mcvm["Minecraft VM<br/>world tarballs<br/>daily 03:30"]
        opssrc["ops self-config<br/>hourly"]
        vms["VMs<br/>jarvis · fr24 · minecraft"]
        win["pc1 · pc2 · tablet<br/>Phase 2"]
    end

    kopia["Kopia server on ops<br/>:51515 · TLS<br/>client-side encrypted"]
    vz["vzdump · nightly 02:00<br/>zstd · keep 7d + 4w"]

    subgraph nas["Synology NAS (+ NAS snapshots = immutability layer)"]
        repo[("kopia/ repository<br/>24h · 7d · 4w · 12m")]
        dumps[("dump/<br/>VM archives")]
    end

    gd[("Google Drive<br/>latest dump set ONLY")]
    b2[("Backblaze B2<br/>planned — repo off-site")]

    mbp --> kopia
    mcvm --> kopia
    opssrc --> kopia
    win -.-> kopia
    kopia ==> repo
    vms --> vz ==> dumps
    dumps -- "04:00 rclone" --> gd
    repo -.-> b2

    classDef planned stroke-dasharray: 5 5,opacity:0.6;
    class win,b2 planned;

The incident that shaped this design

For 62 nights (June–August 2026) the previous setup — Proxmox Backup Server with its datastore on a Synology NFS export — failed every single backup: 186 failed tasks, zero successes, an empty chunk store, while every dashboard and the nightly Google Drive sync looked green. Two permanent rules came out of it: no PBS datastore on NFS, ever (VM backups now use vzdump directly to the share), and monitoring checks content, not exit codes. The full post-mortem lives in security history.

What runs where, on what schedule

When What Where it runs
02:00 nightly vzdump of jarvis, fr24, minecraft → NAS NFS (/mnt/pve/pbs/dump), zstd, keep 7 daily + 4 weekly PVE host (/etc/pve/jobs.cfg)
03:00 nightly Minecraft world tarballs → VM-local ~/bedrock_backups (mc-admin's restore source), pruned to 14 daily + 8 weekly Minecraft VM
03:30 nightly Kopia snapshot of bedrock_backups → NAS repo as backup@mc, keep 30 daily + 52 weekly Minecraft VM
hourly :05 Kopia snapshot of ops /etc + /usr/local/bin (self-backup, keeps the repo warm) ops
hourly MacBook: Kopia snapshot of Documents, Desktop, Pictures — while awake; works remotely over Tailscale MacBook (KopiaUI)
04:00 nightly Off-site: latest vzdump set only → Google Drive (rclone) ops
hourly :35 Health check — pages ntfy only on new problems ops
~09:00 UK Daily digest to ntfy — always sends ops
continuous Kopia repository server for all endpoints (192.168.1.190:51515, TLS, per-machine users) ops
gated OFF Wake-on-LAN timers for the gaming PCs (WOL_ENABLED=0 until Veeam exists) ops
Phase 2 Veeam entire-computer images of the Windows machines → NAS SMB share each PC
NAS UI NAS-side snapshots of the backup shares (immutability layer) NAS

The overnight sequence, visually (each stage finishes before the next needs it):

gantt
    dateFormat HH:mm
    axisFormat %H:%M
    section VM images
    vzdump jarvis + fr24 + minecraft → NAS   :02:00, 40m
    section Minecraft
    world tarballs + local prune (no-touch window)  :03:00, 30m
    kopia snapshot → NAS repo                :03:30, 10m
    section Off-site
    rclone latest dump set → Google Drive    :04:00, 60m
    section Your phone
    daily digest (always arrives)            :09:00, 10m

Off-site gap, by choice

Google Drive holds the latest VM dump set only. The Kopia repo (MacBook + Minecraft history) lives on the NAS with no cloud copy — deliberate for now; Backblaze B2 at roughly $0.40/month is the standing fix. Details and the decision log: off-site.

Everything in the Kopia repo is encrypted before it touches the NAS. One vaulted repo password unlocks it — without that password the repo (and any future cloud copy of it) is unreadable. Secret locations are listed in security posture; no values live in this wiki.

Central config: the one-file NAS swap

Every script and doc takes NAS host, paths, and limits from a single file on ops: /etc/backup-stack/backup-targets.env. When the Synology is replaced by the UNAS Pro 4, the migration is:

  1. Create the export/share + snapshots on the new NAS and copy the dump/ and kopia/ trees across (the Kopia repo holds history and should be copied; dumps age out in 7 days anyway).
  2. Edit the NAS_* values in backup-targets.env.
  3. Run /usr/local/bin/backup-stack-repoint.sh — it verifies what it can and prints exact commands for the steps that can't be abstracted (PVE storage.cfg mount line, Veeam job destinations, NAS-side share setup).
  4. Confirm the next morning's digest is green.

Onboarding a new machine is the same philosophy — one command on ops (add-backup-endpoint.sh <name>) does the server side. See onboarding.

Monitoring philosophy

The monitor never trusts a job's exit code. It checks content: file ages on the NAS, completed snapshot timestamps (interrupted checkpoint records never count as backups), and real byte counts on the remote. A green job that moved zero bytes pages — that rule is the PBS incident's legacy.

Alert semantics, redesigned August 2026:

  • Page (priority high) — fires once when a NEW problem appears: missing or suspiciously small dumps, failed units, unreachable NAS, off-site failure. A persisting problem does not re-page hourly; one "recovered" note is sent when the pageable set clears.
  • Digest-only — stale sources (a laptop closed for a day) and first-backups-that-never-completed appear only in the daily summary, never as pages.
  • Daily digest, ~09:00 UK — one line per component, by guest name (jarvis, fr24, minecraft, macbook — never VMIDs), arrives every day.

Silence is the alarm

The digest always sends. If it stops arriving, the monitor itself is down — investigate ops. See unreachable.

The monitor also warns 30 days before the ops Tailscale key would expire. Manual run from the PVE host: pct exec 104 -- /usr/local/bin/backup-monitor.sh.

Google Drive trash counts against quota

The off-site sync runs with --drive-use-trash=false --delete-before because files in Drive's Trash count against quota for 30 days — trashed prunes wedged the sync entirely once (August 2026, 403 quota errors). Never remove those flags.

Pausing backups (gaming / big test sessions)

  • One machine (preferred): KopiaUI tray icon → Pause All Snapshots. It catches up on the next hourly tick after resuming.
  • Everything at once: on ops, systemctl stop kopia-server.service — endpoints fail soft and retry hourly. systemctl start kopia-server.service to resume.
  • vzdump and Veeam are nightly jobs and never compete with daytime sessions.

Where to go next