Backup Monitoring

Get notified when backups succeed or fail. Works with pg_dump, mysqldump, restic, borgbackup, and any other backup tool.

Backups run silently in the background. You only find out they've been failing when you actually need to restore. Add notifications and catch issues before they matter.

PostgreSQL backup with notifications

backup-postgres.sh

#!/bin/bash
WEBHOOK="https://app.alphorn.dev/api/webhooks/wh_abc123"
DB_NAME="myapp"
BACKUP_DIR="/backups"
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}_$(date +%Y%m%d_%H%M%S).sql.gz"
START_TIME=$(date +%s)

# Run backup
pg_dump -U postgres "$DB_NAME" | gzip > "$BACKUP_FILE"
EXIT_CODE=$?
END_TIME=$(date +%s)
DURATION=$(( END_TIME - START_TIME ))

if [ $EXIT_CODE -eq 0 ]; then
  SIZE=$(du -sh "$BACKUP_FILE" | cut -f1)
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Backup completed: $DB_NAME\",
      \"message\": \"Size: $SIZE, Duration: ${DURATION}s\nFile: $BACKUP_FILE\",
      \"priority\": 2,
      \"tags\": [\"backup\", \"database\", \"success\"]
    }"
else
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Backup FAILED: $DB_NAME\",
      \"message\": \"pg_dump exited with code $EXIT_CODE after ${DURATION}s\",
      \"priority\": 5,
      \"tags\": [\"backup\", \"database\", \"failure\"]
    }"
fi

# Clean up old backups (keep last 7)
ls -t "$BACKUP_DIR"/${DB_NAME}_*.sql.gz | tail -n +8 | xargs rm -f

MySQL backup

backup-mysql.sh

#!/bin/bash
WEBHOOK="https://app.alphorn.dev/api/webhooks/wh_abc123"
DB_NAME="myapp"
BACKUP_FILE="/backups/${DB_NAME}_$(date +%Y%m%d).sql.gz"

mysqldump -u root "$DB_NAME" | gzip > "$BACKUP_FILE"

if [ $? -eq 0 ]; then
  SIZE=$(du -sh "$BACKUP_FILE" | cut -f1)
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{\"title\":\"MySQL backup OK: $DB_NAME\",\"message\":\"Size: $SIZE\",\"priority\":2,\"tags\":[\"backup\",\"success\"]}"
else
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{\"title\":\"MySQL backup FAILED: $DB_NAME\",\"priority\":5,\"tags\":[\"backup\",\"failure\"]}"
fi

Backup size anomaly detection

Alert if a backup is significantly smaller than usual (possible data loss):

EXPECTED_MIN_MB=100
ACTUAL_MB=$(du -m "$BACKUP_FILE" | cut -f1)

if [ "$ACTUAL_MB" -lt "$EXPECTED_MIN_MB" ]; then
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Backup size anomaly: $DB_NAME\",
      \"message\": \"Backup is only ${ACTUAL_MB}MB (expected at least ${EXPECTED_MIN_MB}MB). Possible data loss.\",
      \"priority\": 5,
      \"tags\": [\"backup\", \"anomaly\"]
    }"
fi

Restic

backup-restic.sh

#!/bin/bash
WEBHOOK="https://app.alphorn.dev/api/webhooks/wh_abc123"
REPO="/backups/restic-repo"
PATHS="/home /etc /var/lib"
START_TIME=$(date +%s)

OUTPUT=$(restic -r "$REPO" backup $PATHS 2>&1)
EXIT_CODE=$?
END_TIME=$(date +%s)
DURATION=$(( END_TIME - START_TIME ))

if [ $EXIT_CODE -eq 0 ]; then
  # Extract snapshot summary from restic output
  SUMMARY=$(echo "$OUTPUT" | tail -3)
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Restic backup completed\",
      \"message\": \"Duration: ${DURATION}s\n$SUMMARY\",
      \"priority\": 2,
      \"tags\": [\"backup\", \"restic\", \"success\"]
    }"
else
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Restic backup FAILED\",
      \"message\": \"Exit code $EXIT_CODE after ${DURATION}s\n$OUTPUT\",
      \"priority\": 5,
      \"tags\": [\"backup\", \"restic\", \"failure\"]
    }"
fi

Run retention pruning after backup and notify on failure:

restic -r "$REPO" forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
if [ $? -ne 0 ]; then
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{\"title\":\"Restic prune FAILED\",\"priority\":5,\"tags\":[\"backup\",\"restic\",\"failure\"]}"
fi

Borgbackup

backup-borg.sh

#!/bin/bash
WEBHOOK="https://app.alphorn.dev/api/webhooks/wh_abc123"
REPO="/backups/borg-repo"
ARCHIVE="$REPO::$(hostname)-$(date +%Y%m%d_%H%M%S)"
PATHS="/home /etc /var/lib"
START_TIME=$(date +%s)

OUTPUT=$(borg create --stats "$ARCHIVE" $PATHS 2>&1)
EXIT_CODE=$?
END_TIME=$(date +%s)
DURATION=$(( END_TIME - START_TIME ))

if [ $EXIT_CODE -eq 0 ]; then
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Borg backup completed\",
      \"message\": \"Duration: ${DURATION}s\n$OUTPUT\",
      \"priority\": 2,
      \"tags\": [\"backup\", \"borg\", \"success\"]
    }"
else
  curl -s -X POST "$WEBHOOK" \
    -H "Content-Type: application/json" \
    -d "{
      \"title\": \"Borg backup FAILED\",
      \"message\": \"Exit code $EXIT_CODE after ${DURATION}s\n$OUTPUT\",
      \"priority\": 5,
      \"tags\": [\"backup\", \"borg\", \"failure\"]
    }"
fi

# Prune old archives
borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 "$REPO"

Routing examples

Channel	Filter	Purpose
Slack (#ops)	`tags CONTAINS "failure" AND tags CONTAINS "backup"`	Failed backups only
PagerDuty	`tags CONTAINS "anomaly"`	Size anomalies need immediate attention
Email	`tags CONTAINS "backup"`	Archive all backup activity