diff --git a/README.md b/README.md index ab8dd99..c1301db 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,19 @@ # baktainer Easily backup databases running in docker containers. + ## Features -- Backup MySQL, PostgreSQL, MongoDB, and SQLite databases -- Run on a schedule using cron expressions -- Backup databases running in docker containers -- Define which databases to backup using docker labels +- **Database Support**: MySQL, PostgreSQL, MongoDB, and SQLite databases +- **Scheduled Backups**: Run on a schedule using cron expressions +- **Container Discovery**: Define which databases to backup using docker labels +- **Health Monitoring**: Web dashboard and REST API for monitoring backup status +- **Notifications**: Multi-channel notifications (Slack, Discord, Teams, Email, Webhooks) +- **Backup Rotation**: Automatic cleanup based on age, count, or disk space +- **Encryption**: AES-256-GCM encryption for backup files +- **Compression**: Gzip compression to reduce backup file sizes +- **Performance Monitoring**: Real-time metrics and performance alerts +- **Label Validation**: Schema-based validation with helpful error messages +- **SSL/TLS Support**: Secure Docker API connections +- **High Performance**: Multi-threaded backups with dynamic scaling ## Installation ⚠️ **Security Notice**: Baktainer requires Docker socket access which grants significant privileges. Please review [SECURITY.md](SECURITY.md) for important security considerations and recommended mitigations. @@ -15,35 +24,91 @@ services: image: jamez001/baktainer:latest container_name: baktainer restart: unless-stopped + ports: + - "8080:8080" # Health check dashboard volumes: - ./backups:/backups - /var/run/docker.sock:/var/run/docker.sock environment: - BT_CRON="0 0 * * *" # Backup every day at midnight - - "BT_DOCKER_URL=unix:///var/run/docker.sock" + - BT_DOCKER_URL=unix:///var/run/docker.sock - BT_THREADS=4 - BT_LOG_LEVEL=info - # Enable if using SSL over tcp - #- BT_SSL = true - #- BT_CA - #- BT_CERT - #- BT_KEY + - BT_HEALTH_SERVER_ENABLED=true # Enable health check server + - BT_COMPRESS=true # Enable compression + - BT_ROTATION_ENABLED=true # Enable backup rotation + - BT_RETENTION_DAYS=30 # Keep backups for 30 days + # SSL Configuration (if using remote Docker API) + #- BT_SSL=true + #- BT_CA=/path/to/ca.pem + #- BT_CERT=/path/to/cert.pem + #- BT_KEY=/path/to/key.pem + # Notification Configuration + #- BT_NOTIFICATION_CHANNELS=slack,log + #- BT_SLACK_WEBHOOK_URL=https://hooks.slack.com/... + #- BT_NOTIFY_FAILURES=true + # Encryption Configuration + #- BT_ENCRYPTION_ENABLED=true + #- BT_ENCRYPTION_KEY=your-hex-key ``` For enhanced security, consider using a Docker socket proxy. See [SECURITY.md](SECURITY.md) for detailed security recommendations. ## Environment Variables + +### Core Configuration | Variable | Description | Default | | -------- | ----------- | ------- | -| BT_CRON | Cron expression for scheduling backups | 0 0 * * * | -| BT_THREADS | Number of threads to use for backups | 4 | -| BT_LOG_LEVEL | Log level (debug, info, warn, error) | info | -| BT_COMPRESS | Enable gzip compression for backups | true | -| BT_SSL | Enable SSL for docker connection | false | -| BT_CA | Path to CA certificate | none | -| BT_CERT | Path to client certificate | none | -| BT_KEY | Path to client key | none | -| BT_DOCKER_URL | Docker URL | unix:///var/run/docker.sock | +| BT_CRON | Cron expression for scheduling backups | `0 0 * * *` | +| BT_THREADS | Number of threads to use for backups | `4` | +| BT_LOG_LEVEL | Log level (debug, info, warn, error) | `info` | +| BT_DOCKER_URL | Docker API URL | `unix:///var/run/docker.sock` | +| BT_BACKUP_DIR | Directory to store backups | `/backups` | + +### Backup Options +| Variable | Description | Default | +| -------- | ----------- | ------- | +| BT_COMPRESS | Enable gzip compression for backups | `true` | +| BT_ROTATION_ENABLED | Enable automatic backup rotation | `true` | +| BT_RETENTION_DAYS | Days to keep backups (0 = unlimited) | `30` | +| BT_RETENTION_COUNT | Max backups per container (0 = unlimited) | `0` | +| BT_MIN_FREE_SPACE_GB | Minimum free space in GB | `10` | + +### Health Monitoring +| Variable | Description | Default | +| -------- | ----------- | ------- | +| BT_HEALTH_SERVER_ENABLED | Enable health check server | `false` | +| BT_HEALTH_PORT | Port for health check server | `8080` | +| BT_HEALTH_BIND | Bind address for health server | `0.0.0.0` | + +### SSL/TLS Configuration +| Variable | Description | Default | +| -------- | ----------- | ------- | +| BT_SSL | Enable SSL for Docker connection | `false` | +| BT_CA | Path to CA certificate or certificate data | none | +| BT_CERT | Path to client certificate or certificate data | none | +| BT_KEY | Path to client key or key data | none | + +### Encryption +| Variable | Description | Default | +| -------- | ----------- | ------- | +| BT_ENCRYPTION_ENABLED | Enable AES-256-GCM encryption | `false` | +| BT_ENCRYPTION_KEY | Encryption key (hex or base64) | none | +| BT_ENCRYPTION_KEY_FILE | Path to encryption key file | none | +| BT_ENCRYPTION_PASSPHRASE | Passphrase for key derivation | none | + +### Notifications +| Variable | Description | Default | +| -------- | ----------- | ------- | +| BT_NOTIFICATION_CHANNELS | Comma-separated list of channels | `log` | +| BT_NOTIFY_SUCCESS | Notify on successful backups | `false` | +| BT_NOTIFY_FAILURES | Notify on backup failures | `true` | +| BT_NOTIFY_WARNINGS | Notify on warnings | `true` | +| BT_NOTIFY_HEALTH | Notify on health issues | `true` | +| BT_SLACK_WEBHOOK_URL | Slack webhook URL | none | +| BT_DISCORD_WEBHOOK_URL | Discord webhook URL | none | +| BT_TEAMS_WEBHOOK_URL | Microsoft Teams webhook URL | none | +| BT_WEBHOOK_URL | Generic webhook URL | none | ## Usage Add labels to your docker containers to specify which databases to backup. @@ -68,16 +133,26 @@ services: - baktainer.name="MyApp" ``` -## Possible Values for Labels -| Label | Description | -| ----- | ----------- | -| baktainer.backup | Set to true to enable backup for this container | -| baktainer.db.engine | Database engine (mysql, postgres, mongodb, sqlite) | -| baktainer.db.name | Name of the database to backup | -| baktainer.db.user | Username for the database | -| baktainer.db.password | Password for the database | -| baktainer.name | Name of the application (optional). Determines name of sql dump file. | -| baktainer.compress | Enable gzip compression for this container's backups (true/false). Overrides BT_COMPRESS. | +## Docker Labels for Container Configuration + +### Required Labels +| Label | Description | Required For | +| ----- | ----------- | ------------ | +| baktainer.backup | Set to `true` to enable backup for this container | All | +| baktainer.db.engine | Database engine: `mysql`, `postgres`, `mongodb`, `sqlite` | All | +| baktainer.db.name | Name of the database to backup | All | +| baktainer.db.user | Username for the database | MySQL, PostgreSQL, MongoDB | +| baktainer.db.password | Password for the database | MySQL, PostgreSQL, MongoDB | + +### Optional Labels +| Label | Description | Default | +| ----- | ----------- | ------- | +| baktainer.name | Application name for backup files | Database name | +| baktainer.compress | Enable compression for this container (`true`/`false`) | `BT_COMPRESS` value | +| baktainer.encrypt | Enable encryption for this container (`true`/`false`) | `BT_ENCRYPTION_ENABLED` value | +| baktainer.backup.priority | Backup priority: `low`, `normal`, `high`, `critical` | `normal` | +| baktainer.backup.retention.days | Days to keep backups for this container | `BT_RETENTION_DAYS` value | +| baktainer.backup.retention.count | Max backups to keep for this container | `BT_RETENTION_COUNT` value | ## Backup Files The backup files will be stored in the directory specified by the `BT_BACKUP_DIR` environment variable. The files will be named according to the following format: @@ -88,6 +163,103 @@ Where `` is the date of the backup ('YY-MM-DD' format) `` is the By default, backups are compressed with gzip. To disable compression, set `BT_COMPRESS=false` or add `baktainer.compress=false` label to specific containers. +## Health Monitoring & Dashboard + +Baktainer includes a comprehensive health monitoring system with a web dashboard and REST API. + +### Accessing the Dashboard +When `BT_HEALTH_SERVER_ENABLED=true`, visit **http://localhost:8080** for: +- Real-time backup status and metrics +- Container discovery and configuration +- Performance monitoring with auto-refresh +- System health checks and alerts + +### Health Check Endpoints +| Endpoint | Description | +| -------- | ----------- | +| `GET /` | Interactive monitoring dashboard | +| `GET /health` | Health check (200 = healthy, 503 = unhealthy) | +| `GET /status` | Detailed system status and metrics | +| `GET /backups` | Backup history and statistics | +| `GET /containers` | Discovered containers with backup labels | +| `GET /config` | Configuration (credentials sanitized) | +| `GET /metrics` | Prometheus-format metrics | + +### Prometheus Integration +Use the `/metrics` endpoint to integrate with monitoring systems: +```yaml +# Prometheus scrape config +scrape_configs: + - job_name: 'baktainer' + static_configs: + - targets: ['baktainer:8080'] + metrics_path: '/metrics' + scrape_interval: 30s +``` + +## Notifications + +Configure multi-channel notifications for backup events: + +### Supported Channels +- **Slack**: Set `BT_SLACK_WEBHOOK_URL` +- **Discord**: Set `BT_DISCORD_WEBHOOK_URL` +- **Microsoft Teams**: Set `BT_TEAMS_WEBHOOK_URL` +- **Generic Webhook**: Set `BT_WEBHOOK_URL` +- **Logs**: Always available + +### Example Configuration +```yaml +environment: + - BT_NOTIFICATION_CHANNELS=slack,log + - BT_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL + - BT_NOTIFY_FAILURES=true + - BT_NOTIFY_SUCCESS=false + - BT_NOTIFY_WARNINGS=true +``` + +## Backup Encryption + +Secure your backups with AES-256-GCM encryption: + +```yaml +environment: + - BT_ENCRYPTION_ENABLED=true + - BT_ENCRYPTION_KEY=your-256-bit-hex-key + # Or use a key file: + # - BT_ENCRYPTION_KEY_FILE=/path/to/keyfile + # Or derive from passphrase: + # - BT_ENCRYPTION_PASSPHRASE=your-secure-passphrase +``` + +Encrypted files use `.enc` extension and include authentication data for integrity verification. + +## Backup Rotation & Cleanup + +Automatic backup cleanup keeps your storage manageable: + +### Configuration Options +```yaml +environment: + - BT_ROTATION_ENABLED=true # Enable automatic cleanup + - BT_RETENTION_DAYS=30 # Keep backups for 30 days + - BT_RETENTION_COUNT=0 # Max backups per container (0 = unlimited) + - BT_MIN_FREE_SPACE_GB=10 # Trigger cleanup when free space < 10GB +``` + +### Per-Container Overrides +```yaml +labels: + - baktainer.backup.retention.days=7 # Keep this container's backups for 7 days + - baktainer.backup.retention.count=5 # Keep max 5 backups for this container +``` + +### Cleanup Behavior +- **Time-based**: Remove backups older than specified days +- **Count-based**: Keep only N most recent backups per container +- **Space-based**: Automatic cleanup when disk space is low +- **Smart cleanup**: Removes empty date directories after cleanup + ## Testing The project includes comprehensive test coverage with both unit and integration tests. @@ -125,20 +297,32 @@ bin/test --all --coverage bin/test --integration --setup --cleanup ``` -## Roadmap -- [x] Add support for SQLite backups -- [x] Add support for MongoDB backups -- [x] Add support for MySQL backups -- [x] Add support for PostgreSQL backups -- [x] Add support for cron scheduling -- [x] Add support for Docker labels to specify which databases to backup -- [x] Add support for Docker socket -- [x] Add support for Docker API over TCP -- [x] Add support for Docker API over SSL -- [x] Add support for Docker API over HTTP -- [x] Add support for Docker API over HTTPS -- [x] Add support for Docker API over Unix socket -- [x] Add comprehensive test coverage (94.94% line coverage) -- [ ] Add individual hook for completed backups -- [ ] Add hook for fullly completed backups -- [ ] Optionally limit time for each backup +## Development Roadmap + +### ✅ Completed Features +- [x] **Database Support**: MySQL, PostgreSQL, MongoDB, and SQLite backups +- [x] **Scheduling**: Cron-based backup scheduling +- [x] **Container Discovery**: Docker label-based configuration +- [x] **Docker Integration**: Support for socket, TCP, SSL/TLS connections +- [x] **Compression**: Gzip compression for backup files +- [x] **Health Monitoring**: Web dashboard and REST API monitoring +- [x] **Notifications**: Multi-channel notifications (Slack, Discord, Teams, Webhooks) +- [x] **Backup Rotation**: Automatic cleanup based on age, count, and disk space +- [x] **Encryption**: AES-256-GCM encryption for backup files +- [x] **Performance Monitoring**: Real-time metrics and alerts +- [x] **Label Validation**: Schema-based validation with helpful error messages +- [x] **High Performance**: Multi-threaded backups with dynamic scaling +- [x] **Comprehensive Testing**: 94.94% line coverage with unit and integration tests + +### 🔄 In Progress +- [ ] Backup streaming for large databases +- [ ] Advanced retry strategies with exponential backoff + +### 📋 Future Enhancements +- [ ] Individual hooks for completed backups +- [ ] Hooks for fully completed backup cycles +- [ ] Configurable timeout limits for each backup +- [ ] Database-specific optimization settings +- [ ] Backup verification and integrity checking +- [ ] Multi-region backup replication +- [ ] Advanced alerting rules and thresholds