fix: Backup sync failures

[ainsleyclark]

Resolves HTTP 429 rate limit errors during backup synchronization to Google Drive.

Changes:

• Reduced concurrent transfers from 8 to 4 (--transfers, --checkers)
• Added retry logic: --retries 10, --low-level-retries 10, --retries-sleep 10s
• Added rate limiting: --tpslimit 5, --tpslimit-burst 10
• Added API call pacing: --drive-pacer-min-sleep 100ms, --drive-pacer-burst 100
• Increased chunk size from 64M to 256M (fewer API calls for large files)
• Increased job timeout from 2 hours to 3 hours to accommodate rate limiting delays

This should resolve the "googleapi: got HTTP response code 429" errors that occur when syncing large backup files (~19GB) to Google Drive Shared Drives.

[claude]

Review summary

• Overall score: 8/10
• Critical issues: 0
• Warnings: 1
• Suggestions: 2
• Recommendation: ✅ Approve with comments

Well-structured fix that addresses Google Drive API rate limiting through a comprehensive combination of retry logic, rate limiting, and request pacing. The changes are appropriate for resolving HTTP 429 errors during large file synchronisation.

Critical issues 🔴

None

Warnings 🟡

Timeout increase may mask underlying issues

Location: internal/templates/.github/workflows/backup.yaml.tmpl:325

The timeout increase from 2 hours to 3 hours (50% increase) is significant. While this accommodates rate limiting delays, consider:

• Monitoring actual sync duration to ensure 3 hours is sufficient but not excessive
• The timeout might need adjustment based on bucket size growth over time
• Failed jobs will now run 50% longer before timing out, delaying failure notifications

Suggestion: Consider adding workflow metrics or logging to track sync duration patterns, which would help validate whether 3 hours is optimal or needs further tuning.

Suggestions 🟢

Consider documenting rclone flag choices

The combination of rate limiting flags is well-chosen for Google Drive's API constraints, but the specific values (--tpslimit 5, --drive-pacer-min-sleep 100ms, etc.) lack context for future maintainers. Consider adding a brief comment in the template explaining the rationale behind these specific values, particularly since they're tuned for Google Drive Shared Drives with large files (~19GB).

Chunk size increase trade-offs

Increasing chunk size from 64M to 256M reduces API calls but increases memory usage per transfer. With --transfers=4, this means up to 1GB of data in flight simultaneously (4 × 256M). This is likely fine for GitHub Actions runners (7GB RAM), but worth noting for awareness.

Detail: The chunk size increase is a good optimisation for large files, but if backup files vary significantly in size, smaller files won't benefit and might see slightly higher memory overhead. Monitor for any out-of-memory issues in CI logs.

---

Overall assessment: The changes demonstrate a solid understanding of rclone's rate limiting capabilities and Google Drive API constraints. The retry logic is robust, the rate limiting parameters are conservative, and the chunk size optimisation is sensible for the reported file sizes. The PR addresses the stated issue effectively.