ops: nightly DB backup + self-hosted uptime monitoring
CI/CD / CI · API (dotnet build + test) (push) Successful in 41s
CI/CD / CI · Admin API (dotnet build) (push) Successful in 30s
CI/CD / CI · Dashboard (tsc) (push) Successful in 1m10s
CI/CD / CI · Admin Web (tsc) (push) Successful in 37s
CI/CD / CI · Website (tsc) (push) Successful in 44s
CI/CD / CI · Koja (tsc) (push) Successful in 50s
CI/CD / Deploy · all services (push) Successful in 1m48s
CI/CD / CI · API (dotnet build + test) (push) Successful in 41s
CI/CD / CI · Admin API (dotnet build) (push) Successful in 30s
CI/CD / CI · Dashboard (tsc) (push) Successful in 1m10s
CI/CD / CI · Admin Web (tsc) (push) Successful in 37s
CI/CD / CI · Website (tsc) (push) Successful in 44s
CI/CD / CI · Koja (tsc) (push) Successful in 50s
CI/CD / Deploy · all services (push) Successful in 1m48s
Backup (production data-loss protection — was none):
- meezi-backup sidecar in docker-compose.yml runs pg_dump nightly at 02:00
Tehran, gzip, 14-day rotation, atomic .partial→final, into ./backups
(persists across deploys; rsync off-box per RESTORE.md).
- Wired into the deploy job (up -d --no-deps backup); takes one dump on boot.
- scripts/backup/pg-backup-loop.sh + RESTORE.md (restore + off-box guidance).
Monitoring:
- docker-compose.monitoring.yml: Uptime Kuma stack (own volume), stood up
once, independent of app deploys.
- Caddyfile status.{$DOMAIN} route; docs/monitoring.md lists the exact
monitors (incl. /q guest-menu 200 check) + TLS-expiry alerts (catches the
~90-day cert breakage early) + alert-channel setup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Meezi database backup & restore
|
||||
|
||||
## How backups work
|
||||
The `meezi-backup` container (in `docker-compose.yml`) runs a nightly `pg_dump`
|
||||
of the whole `meezi` database at **02:00 Asia/Tehran**, gzips it, and keeps the
|
||||
**last 14 days** in the host `./backups` directory (override with `BACKUP_DIR`).
|
||||
Filenames: `meezi_YYYYMMDD_HHMMSS.sql.gz`. One backup is also taken immediately
|
||||
when the container first starts.
|
||||
|
||||
Check it's running / list backups:
|
||||
```bash
|
||||
docker logs meezi-backup --tail 20
|
||||
ls -lh ./backups
|
||||
```
|
||||
|
||||
## ⚠️ Copy backups OFF the server
|
||||
The bind-mounted `./backups` survives a container/volume wipe, but **not a disk
|
||||
failure**. Add an off-box copy (run from the host via cron), e.g.:
|
||||
```bash
|
||||
# rsync to another host nightly at 03:00
|
||||
0 3 * * * rsync -az --delete /path/to/meezi/backups/ user@backup-host:/srv/meezi-backups/
|
||||
```
|
||||
or `rclone copy ./backups remote:meezi-backups` to object storage.
|
||||
|
||||
## Restore
|
||||
1. Pick a dump:
|
||||
```bash
|
||||
ls -lh ./backups # choose e.g. meezi_20260615_020000.sql.gz
|
||||
```
|
||||
2. (Recommended) stop the API so nothing writes mid-restore:
|
||||
```bash
|
||||
docker stop meezi-api
|
||||
```
|
||||
3. Restore into the running Postgres container:
|
||||
```bash
|
||||
gunzip -c ./backups/meezi_20260615_020000.sql.gz \
|
||||
| docker exec -i meezi-db psql -U meezi -d meezi
|
||||
```
|
||||
For a clean restore into an empty DB, drop & recreate first:
|
||||
```bash
|
||||
docker exec -i meezi-db psql -U meezi -d postgres -c "DROP DATABASE meezi;"
|
||||
docker exec -i meezi-db psql -U meezi -d postgres -c "CREATE DATABASE meezi OWNER meezi;"
|
||||
gunzip -c ./backups/<dump>.sql.gz | docker exec -i meezi-db psql -U meezi -d meezi
|
||||
```
|
||||
4. Start the API again (it runs EF migrations on boot, which is a no-op if the
|
||||
dump is current):
|
||||
```bash
|
||||
docker start meezi-api
|
||||
```
|
||||
|
||||
## Manual one-off backup
|
||||
```bash
|
||||
docker exec meezi-db pg_dump -U meezi --no-owner --no-privileges meezi \
|
||||
| gzip -9 > ./backups/meezi_manual_$(date +%Y%m%d_%H%M%S).sql.gz
|
||||
```
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/bin/sh
|
||||
# Nightly Postgres backup loop for Meezi.
|
||||
#
|
||||
# Runs inside a small postgres-image container (has pg_dump/gzip). Every day at
|
||||
# ~02:00 Tehran it dumps the whole database, gzips it, and keeps the last
|
||||
# RETAIN_DAYS files in /backups. Designed to be dead-simple and dependency-free:
|
||||
# no cron daemon, just sleep-until-next-run so it survives container restarts.
|
||||
#
|
||||
# Env:
|
||||
# PGHOST, PGUSER, PGPASSWORD, PGDATABASE — connection (from compose)
|
||||
# RETAIN_DAYS — how many daily dumps to keep (default 14)
|
||||
# BACKUP_HOUR — local hour to run (default 2 = 02:00)
|
||||
set -eu
|
||||
|
||||
RETAIN_DAYS="${RETAIN_DAYS:-14}"
|
||||
BACKUP_HOUR="${BACKUP_HOUR:-2}"
|
||||
OUT_DIR=/backups
|
||||
export TZ="${TZ:-Asia/Tehran}"
|
||||
|
||||
log() { echo "[pg-backup $(date '+%Y-%m-%d %H:%M:%S %Z')] $*"; }
|
||||
|
||||
run_backup() {
|
||||
ts=$(date '+%Y%m%d_%H%M%S')
|
||||
tmp="$OUT_DIR/.meezi_${ts}.sql.gz.partial"
|
||||
final="$OUT_DIR/meezi_${ts}.sql.gz"
|
||||
log "starting dump → $final"
|
||||
# pg_dump streams to gzip; .partial then atomic rename so a crash never
|
||||
# leaves a truncated file that looks like a good backup.
|
||||
if pg_dump --no-owner --no-privileges | gzip -9 > "$tmp"; then
|
||||
mv "$tmp" "$final"
|
||||
size=$(wc -c < "$final" 2>/dev/null || echo '?')
|
||||
log "done ($size bytes)"
|
||||
else
|
||||
rm -f "$tmp"
|
||||
log "ERROR: dump failed"
|
||||
return 1
|
||||
fi
|
||||
# Rotate: delete dumps older than RETAIN_DAYS days.
|
||||
find "$OUT_DIR" -maxdepth 1 -name 'meezi_*.sql.gz' -mtime "+${RETAIN_DAYS}" -print -delete | while read -r f; do
|
||||
log "rotated out $f"
|
||||
done
|
||||
}
|
||||
|
||||
seconds_until_next_run() {
|
||||
now_h=$(date '+%-H'); now_m=$(date '+%-M'); now_s=$(date '+%-S')
|
||||
now=$(( now_h * 3600 + now_m * 60 + now_s ))
|
||||
target=$(( BACKUP_HOUR * 3600 ))
|
||||
if [ "$now" -lt "$target" ]; then
|
||||
echo $(( target - now ))
|
||||
else
|
||||
echo $(( 86400 - now + target ))
|
||||
fi
|
||||
}
|
||||
|
||||
log "backup loop started (retain ${RETAIN_DAYS}d, daily at ${BACKUP_HOUR}:00 ${TZ})"
|
||||
# Take one backup immediately on first boot so we never sit a full day with none.
|
||||
run_backup || true
|
||||
while true; do
|
||||
wait_s=$(seconds_until_next_run)
|
||||
log "next backup in ${wait_s}s"
|
||||
sleep "$wait_s"
|
||||
run_backup || true
|
||||
done
|
||||
Reference in New Issue
Block a user