How to effectively monitor regular backups

Source: Dev.to

Scenario

Imagine this scenario: you wrote a Bash script to back up a production database (e.g., an online store). After adding the script to crontab, everything worked flawlessly. A month later the database becomes corrupted—perhaps because a faulty plugin was installed. When you try to restore from the “latest” backup, you discover the most recent backup is two weeks old.

What happened?
The backup script silently stopped working. This nightmare is more common than you think and can be caused by:

• Full disk
• Permission changes
• Network timeouts
• Expired credentials
• Typos introduced during a quick fix

The Problem with Unmonitored Backups

Traditional cron jobs have a fundamental flaw: they only report an error when they fail to run. A backup script can:

• Exit with errors while still “succeeding” from cron’s point of view
• Produce empty or corrupted files
• Take far longer than expected (a sign of underlying issues)
• Skip tables due to permission problems

Before you know it, the retention period expires and you’re left without any usable backups.

Monitoring Backup Scripts

The solution is simple: have the backup script actively report its status to an external monitor. Below is a guide on integrating database backups with CronMonitor.

MySQL / MariaDB Backup Example

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/mysql"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

# Signal start
curl -s "${MONITOR_URL}/start"

# Perform backup
mysqldump --single-transaction \
    --routines \
    --triggers \
    "$DB_NAME" | gzip > "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz"

# Verify backup succeeded and file is not empty
if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.sql.gz" ]; then
    # Keep only the last 7 days
    find "$BACKUP_DIR" -name "*.sql.gz" -mtime +7 -delete

    # Signal success
    curl -s "${MONITOR_URL}/complete"
else
    # Signal failure
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

PostgreSQL Backup Example

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups/postgres"
DATE=$(date +%Y%m%d_%H%M%S)
DB_NAME="production"

# Signal start
curl -s "${MONITOR_URL}/start"

# Perform backup (custom format for flexibility)
pg_dump -Fc "$DB_NAME" > "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump"

# Verify backup succeeded and file is not empty
if [ $? -eq 0 ] && [ -s "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" ]; then
    # Verify integrity
    pg_restore --list "${BACKUP_DIR}/${DB_NAME}_${DATE}.dump" > /dev/null 2>&1
    if [ $? -eq 0 ]; then
        curl -s "${MONITOR_URL}/complete"
    else
        curl -s "${MONITOR_URL}/fail"
        exit 1
    fi
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

Multi‑Database Backup with Size Reporting

#!/bin/bash

MONITOR_URL="https://cronmonitor.app/api/ping/your-unique-id"
BACKUP_DIR="/backups"
DATE=$(date +%Y%m%d)
DATABASES="app_production analytics users"

# Signal start
curl -s "${MONITOR_URL}/start"

FAILED=0
TOTAL_SIZE=0

for DB in $DATABASES; do
    mysqldump --single-transaction "$DB" | gzip > "${BACKUP_DIR}/${DB}_${DATE}.sql.gz"

    if [ $? -ne 0 ] || [ ! -s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" ]; then
        FAILED=1
        echo "Backup failed for: $DB"
    else
        # Get file size (compatible with macOS and Linux)
        SIZE=$(stat -f%z "${BACKUP_DIR}/${DB}_${DATE}.sql.gz" 2>/dev/null || stat -c%s "${BACKUP_DIR}/${DB}_${DATE}.sql.gz")
        TOTAL_SIZE=$((TOTAL_SIZE + SIZE))
    fi
done

if [ $FAILED -eq 0 ]; then
    # Report success with total size metadata
    curl -s "${MONITOR_URL}/complete?msg=Backed%20up%20${TOTAL_SIZE}%20bytes"
else
    curl -s "${MONITOR_URL}/fail"
    exit 1
fi

How to Configure CronMonitor for a Backup Job

1. Create a new monitor in CronMonitor for the “backup” job.
2. Set the projected schedule (e.g., daily at 02:00 AM).
3. Configure a grace period—long enough for the largest expected backup.
4. Set up alerts (email, Slack, Discord, etc.).

Key Settings

• Schedule – Must match the server’s cron schedule exactly.
• Grace period – Longer than the longest projected backup time.

Best Practices

1. Verify, don’t assume
   Always check that the backup file exists and contains data. An empty gzipped file is still a “successful” command from the shell’s perspective.

2. Test restores regularly
   Backups are only as good as the ability to restore them. Schedule periodic restore tests and monitor those jobs as well.

3. Monitor backup duration
   CronMonitor records how long each job takes. A sudden increase often signals growing data volume or performance problems.

4. Store backups off‑site
   Include a synchronization step to an external drive or server.

   # After local backup succeeds
   rsync -az "${BACKUP_DIR}/" remote:/backups/ && \
       curl -s "${MONITOR_URL}/complete" || \
       curl -s "${MONITOR_URL}/fail"

5. Document recovery procedures
   If an alert fires (e.g., a Slack notification about a failed backup), you need clear, step‑by‑step instructions—not a debugging session.

Conclusion

Database backups are the last line of defense against data loss. They deserve more than a fire‑and‑forget cron job and the hope that everything will always run correctly. By adding active monitoring you get immediate visibility into problems, allowing you to act before it’s too late.

While you still have time to resolve it.

Start monitoring your backup scripts today. Your future self (the one who doesn’t have to explain data loss to a client) will thank you.

CronMonitor is a simple, developer‑friendly cron job monitoring service. Set up your first monitor in under a minute.