feat: implement memory-optimized backup system with large database support v0.2.0
Major performance and reliability improvements for large database backups
- Replace Docker API with direct shell execution using Open3.popen3
- Implement 4KB chunk streaming to minimize memory usage
- Add aggressive garbage collection every 10MB
- Eliminate stderr accumulation in memory
- Reduce memory usage from 1.9GB to <100MB for large backups
- Add PostgreSQL timeout protection via BT_PG_LOCK_TIMEOUT (default: 60s)
- Add PostgreSQL statement timeout via BT_PG_STATEMENT_TIMEOUT (default: 0)
- Add backup process timeout via BT_BACKUP_TIMEOUT (default: 30min)
- Add stall detection (5 minutes no output)
- Add --lock-wait-timeout parameter for pg_dump commands
- Capture actual exit codes from backup commands
- Implement comprehensive timeout and stall detection
- Add progress logging every 100MB
- Improve error reporting with detailed debugging
- Add complete development environment with sample databases
- Add dry-run mode for configuration validation
- Add development setup scripts and docker-compose.dev.yml
- Add DEVELOPMENT.md and TROUBLESHOOTING.md guides
- Add Docker CLI to container for shell execution
- Update all PostgreSQL tests for new timeout parameters
- Maintain backward compatibility with existing configurations
- Add comprehensive test coverage (629 unit tests, 0 failures)
Fixes #high-memory-usage-large-databases
james released this