Security history¶
This page is the institutional memory: the incidents that actually happened, what they cost, and the permanent rules they produced. When a future change proposal seems to conflict with one of these rules, read the incident before overriding the rule. Current state lives in Security posture.
| Date | Incident | Severity |
|---|---|---|
| 2026-06-08 → 2026-08-08 | PBS-over-NFS: 62 nights of green, zero backups | Critical |
| 2026-08-11 | Checkpoint counted as a fresh backup | Near-miss |
| 2026-08-11 | Google Drive Trash quota outage | Moderate |
Incident 1 — PBS over NFS¶
Date: 2026-06-08 to 2026-08-08 (62 nights)
Impact: Zero recoverable VM backups for the entire period. 186 failed PBS tasks, zero successes, an empty chunk store — while every dashboard showed green and the nightly Google Drive sync dutifully mirrored the empty datastore off-site each morning. Had a VM died in that window, there was nothing to restore. The June 2026 audit had flagged the warning sign (finding 4c: images present but never verify-jobbed or test-restored) two days into the failure window.
Cause: The PBS datastore lived on a Synology NFS export. PBS datastore semantics are incompatible with Synology's NFS implementation, so every backup task failed after the job scheduler had already recorded a run. The monitoring of the day watched exit codes and job schedules, and the off-site sync copied whatever was there — nobody ever asked whether actual backup bytes existed.
Fix: PBS decommissioned entirely (LXC 101 stopped 2026-08-09,
onboot=0, destroy after 2026-09-09). VM backups rebuilt as plain nightly
vzdump direct to the NAS share — a mechanism that is just files, whose
success is trivially checkable by looking at the files. Monitoring rebuilt to
check content. See Backups overview.
Permanent rule 1 — no PBS datastore on NFS, ever
If PBS (or any chunk-store backup product) ever returns, its datastore goes on a local disk or a block device — never on a Synology/UNAS NFS export. vzdump-to-share works because it writes ordinary files.
Permanent rule 2 — monitor content, not exit codes
A green job that moved zero bytes must page. The monitor checks file ages, snapshot timestamps, and remote byte counts — never "did the unit exit 0". This rule is why the current monitor exists in its present form.
Lesson: A backup system's health signal must be derived from the thing you would restore, not from the machinery that produces it. Sixty-two consecutive green dashboards proved nothing except that the dashboard was measuring the wrong layer.
Incident 2 — checkpoint counted as a fresh backup¶
Date: 2026-08-11
Impact: Near-miss, caught during monitor shakedown. The health check treated a kopia checkpoint record — the marker kopia writes partway through a long-running snapshot — as a completed backup. A source whose snapshot had started but never finished would have shown as fresh indefinitely. No data was lost; the flaw was in the reporting, exactly where the PBS incident lived.
Cause: The monitor's freshness query listed kopia manifests without filtering out incomplete/checkpoint entries. A checkpoint proves a snapshot started; it proves nothing about restorable content.
Fix: Monitor amended the same day: checkpoint records never count as backups. Relatedly, never-completed first backups from new endpoints are digest-only rather than silently absent — a machine that has never finished a snapshot is visibly listed as such in the 09:00 digest.
Lesson: Rule 2 from the PBS incident, in miniature — and proof that the rule needs re-applying at every layer. Even a content-based monitor can pick the wrong artifact as its evidence. "A backup exists" means a completed snapshot with content, not any record with a recent timestamp.
Incident 3 — Google Drive Trash quota outage¶
Date: 2026-08-11
Impact: Off-site sync failed with quota-exhaustion 403s. The nightly vzdump set could not upload, so the off-site copy went stale until fixed. Alert fired (per rule 2, the monitor watches remote byte counts, so this one was caught rather than silent).
Cause: rclone's default delete behaviour moves remote files to Google Drive's Trash — and trashed items count against quota for 30 days. Each night's prune of the old dump set (28.6 GB class of data) accumulated in Trash instead of freeing space, until the quota ran out.
Fix: The sync runs with --drive-use-trash=false --delete-before —
deletions are immediate and space is freed before the new set uploads.
Never remove those flags. The already-trashed 2026-08-09 set self-purges
around 2026-09-08; rclone cleanup gdrive: (on ops) empties Drive Trash
sooner if the space is wanted — operator's call, since it is irreversible.
See Off-site.
Lesson: Know the delete semantics of every remote. On Google Drive, "deleted" means "still consuming your quota for a month". Any future remote (B2 included) gets the same question asked before it carries prunes: what does a delete actually free, and when?
Audit trail¶
| Date | Scope | Record |
|---|---|---|
| 2026-06-10 | Full home-infra + Minecraft audit (VPS, MC VM, PVE, PBS, public hostnames) | Findings and their current status are cross-checked in Security posture |
| 2026-08-12 | Four-lens backup-stack audit (ops/kopia, pve/NFS, VPS, tailnet/artifacts) | Fixed-in-pass items and the pending operator list live in Security posture |