ops: nightly DB backup + self-hosted uptime monitoring
CI/CD / CI · API (dotnet build + test) (push) Successful in 41s
CI/CD / CI · Admin API (dotnet build) (push) Successful in 30s
CI/CD / CI · Dashboard (tsc) (push) Successful in 1m10s
CI/CD / CI · Admin Web (tsc) (push) Successful in 37s
CI/CD / CI · Website (tsc) (push) Successful in 44s
CI/CD / CI · Koja (tsc) (push) Successful in 50s
CI/CD / Deploy · all services (push) Successful in 1m48s

Backup (production data-loss protection — was none):
- meezi-backup sidecar in docker-compose.yml runs pg_dump nightly at 02:00
  Tehran, gzip, 14-day rotation, atomic .partial→final, into ./backups
  (persists across deploys; rsync off-box per RESTORE.md).
- Wired into the deploy job (up -d --no-deps backup); takes one dump on boot.
- scripts/backup/pg-backup-loop.sh + RESTORE.md (restore + off-box guidance).

Monitoring:
- docker-compose.monitoring.yml: Uptime Kuma stack (own volume), stood up
  once, independent of app deploys.
- Caddyfile status.{$DOMAIN} route; docs/monitoring.md lists the exact
  monitors (incl. /q guest-menu 200 check) + TLS-expiry alerts (catches the
  ~90-day cert breakage early) + alert-channel setup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
soroush.asadi
2026-06-15 18:45:07 +03:30
parent d407f0b3e9
commit 32a7cf5b25
7 changed files with 231 additions and 0 deletions
+55
View File
@@ -0,0 +1,55 @@
# Meezi database backup & restore
## How backups work
The `meezi-backup` container (in `docker-compose.yml`) runs a nightly `pg_dump`
of the whole `meezi` database at **02:00 Asia/Tehran**, gzips it, and keeps the
**last 14 days** in the host `./backups` directory (override with `BACKUP_DIR`).
Filenames: `meezi_YYYYMMDD_HHMMSS.sql.gz`. One backup is also taken immediately
when the container first starts.
Check it's running / list backups:
```bash
docker logs meezi-backup --tail 20
ls -lh ./backups
```
## ⚠️ Copy backups OFF the server
The bind-mounted `./backups` survives a container/volume wipe, but **not a disk
failure**. Add an off-box copy (run from the host via cron), e.g.:
```bash
# rsync to another host nightly at 03:00
0 3 * * * rsync -az --delete /path/to/meezi/backups/ user@backup-host:/srv/meezi-backups/
```
or `rclone copy ./backups remote:meezi-backups` to object storage.
## Restore
1. Pick a dump:
```bash
ls -lh ./backups # choose e.g. meezi_20260615_020000.sql.gz
```
2. (Recommended) stop the API so nothing writes mid-restore:
```bash
docker stop meezi-api
```
3. Restore into the running Postgres container:
```bash
gunzip -c ./backups/meezi_20260615_020000.sql.gz \
| docker exec -i meezi-db psql -U meezi -d meezi
```
For a clean restore into an empty DB, drop & recreate first:
```bash
docker exec -i meezi-db psql -U meezi -d postgres -c "DROP DATABASE meezi;"
docker exec -i meezi-db psql -U meezi -d postgres -c "CREATE DATABASE meezi OWNER meezi;"
gunzip -c ./backups/<dump>.sql.gz | docker exec -i meezi-db psql -U meezi -d meezi
```
4. Start the API again (it runs EF migrations on boot, which is a no-op if the
dump is current):
```bash
docker start meezi-api
```
## Manual one-off backup
```bash
docker exec meezi-db pg_dump -U meezi --no-owner --no-privileges meezi \
| gzip -9 > ./backups/meezi_manual_$(date +%Y%m%d_%H%M%S).sql.gz
```
+63
View File
@@ -0,0 +1,63 @@
#!/bin/sh
# Nightly Postgres backup loop for Meezi.
#
# Runs inside a small postgres-image container (has pg_dump/gzip). Every day at
# ~02:00 Tehran it dumps the whole database, gzips it, and keeps the last
# RETAIN_DAYS files in /backups. Designed to be dead-simple and dependency-free:
# no cron daemon, just sleep-until-next-run so it survives container restarts.
#
# Env:
# PGHOST, PGUSER, PGPASSWORD, PGDATABASE — connection (from compose)
# RETAIN_DAYS — how many daily dumps to keep (default 14)
# BACKUP_HOUR — local hour to run (default 2 = 02:00)
set -eu
RETAIN_DAYS="${RETAIN_DAYS:-14}"
BACKUP_HOUR="${BACKUP_HOUR:-2}"
OUT_DIR=/backups
export TZ="${TZ:-Asia/Tehran}"
log() { echo "[pg-backup $(date '+%Y-%m-%d %H:%M:%S %Z')] $*"; }
run_backup() {
ts=$(date '+%Y%m%d_%H%M%S')
tmp="$OUT_DIR/.meezi_${ts}.sql.gz.partial"
final="$OUT_DIR/meezi_${ts}.sql.gz"
log "starting dump → $final"
# pg_dump streams to gzip; .partial then atomic rename so a crash never
# leaves a truncated file that looks like a good backup.
if pg_dump --no-owner --no-privileges | gzip -9 > "$tmp"; then
mv "$tmp" "$final"
size=$(wc -c < "$final" 2>/dev/null || echo '?')
log "done ($size bytes)"
else
rm -f "$tmp"
log "ERROR: dump failed"
return 1
fi
# Rotate: delete dumps older than RETAIN_DAYS days.
find "$OUT_DIR" -maxdepth 1 -name 'meezi_*.sql.gz' -mtime "+${RETAIN_DAYS}" -print -delete | while read -r f; do
log "rotated out $f"
done
}
seconds_until_next_run() {
now_h=$(date '+%-H'); now_m=$(date '+%-M'); now_s=$(date '+%-S')
now=$(( now_h * 3600 + now_m * 60 + now_s ))
target=$(( BACKUP_HOUR * 3600 ))
if [ "$now" -lt "$target" ]; then
echo $(( target - now ))
else
echo $(( 86400 - now + target ))
fi
}
log "backup loop started (retain ${RETAIN_DAYS}d, daily at ${BACKUP_HOUR}:00 ${TZ})"
# Take one backup immediately on first boot so we never sit a full day with none.
run_backup || true
while true; do
wait_s=$(seconds_until_next_run)
log "next backup in ${wait_s}s"
sleep "$wait_s"
run_backup || true
done