A Cloudflare Worker that creates ZIP archives from files stored in Cloudflare R2 buckets. This service processes ZIP creation jobs asynchronously using Cloudflare Queues and includes features like duplicate prevention, streaming compression, and multipart uploads.
- Asynchronous Processing: Uses Cloudflare Queues for background ZIP creation
- Streaming Compression: Never buffers entire files in memory using fflate
- Duplicate Prevention: Uses Durable Objects to prevent concurrent ZIP creation for the same prefix
- Multipart Uploads: Efficiently uploads large ZIPs to R2 using multipart uploads
- Configurable Limits: Set maximum file count and ZIP size limits
- Manifest Generation: Automatically includes a JSON manifest with file statistics
The service consists of:
- HTTP Producer: Accepts ZIP job requests via POST to
/enqueue-zip - Queue Consumer: Processes ZIP jobs in the background
- Durable Object: Manages locks to prevent duplicate work
- R2 Integration: Reads from source bucket, writes to output bucket
- Cloudflare account with Workers and R2 enabled
- Two R2 buckets (source and output)
- Wrangler CLI installed
Set these in your wrangler.toml or via the Cloudflare dashboard:
SOURCE_BUCKET: R2 bucket containing files to zipOUTPUT_BUCKET: R2 bucket where ZIPs will be storedZIP_OUTPUT_PREFIX: Prefix for generated ZIP files (default: "prebaked-zips/")MAX_FILES: Maximum number of files per ZIP (default: 5000)MAX_ZIP_BYTES: Maximum ZIP size in bytes (default: 500MB)
- R2 Buckets: Source and output buckets
- Queue: For job processing (
zip-bundles) - Durable Object: For locking (
ZipLocksDO)
- Clone the repository:
git clone <repository-url>
cd file-zipper- Install dependencies:
npm install-
Configure your
wrangler.tomlwith your bucket names and settings -
Deploy to Cloudflare:
npm run wrangler:deploySend a POST request to /enqueue-zip with the following JSON payload:
{
"prefix": "path/to/files/",
"zipKey": "optional-custom-output-key.zip",
"includeEmpty": true,
"createdBy": "api"
}Parameters:
prefix(required): R2 prefix to collect files fromzipKey(optional): Custom output key in the output bucketincludeEmpty(optional): Include zero-byte files (default: true)createdBy(optional): Audit field for tracking
curl -X POST https://your-worker.your-subdomain.workers.dev/enqueue-zip \
-H "Content-Type: application/json" \
-d '{
"prefix": "uploads/2024/01/",
"zipKey": "january-uploads.zip",
"createdBy": "scheduled-job"
}'- Job Enqueue: HTTP endpoint accepts ZIP job requests and adds them to a queue
- Lock Acquisition: Worker acquires a per-prefix lock using Durable Objects
- File Discovery: Lists all objects with the specified prefix from source bucket
- Streaming ZIP Creation: Creates ZIP using fflate's streaming API
- Multipart Upload: Uploads ZIP to output bucket using multipart uploads
- Manifest Generation: Adds a JSON manifest with file statistics
- Lock Release: Releases the lock when complete
# Login to Cloudflare
npm run wrangler:login
# Deploy to Cloudflare
npm run wrangler:deployfile-zipper/
├── src/
│ └── index.ts # Main worker code
├── package.json # Dependencies and scripts
├── wrangler.toml # Cloudflare configuration
└── README.md # This file
- fflate: Fast ZIP compression library
- @cloudflare/workers-types: TypeScript types for Cloudflare Workers
- wrangler: Cloudflare Workers CLI tool
- Maximum 5000 files per ZIP (configurable)
- Maximum 500MB ZIP size (configurable)
- Files must be in the same R2 bucket
- ZIP creation is asynchronous (not real-time)
- Failed jobs are retried with exponential backoff
- Dead letter queue for permanently failed jobs
- Lock timeouts prevent stuck processes
- Comprehensive error logging
ISC