feat: Bring up separate attachment delivery worker to handle file attachments instead of blocking the middleware #19
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull request overview
This PR introduces a new attachment delivery worker to handle file downloads and delivery asynchronously, preventing the bot from being blocked during large file operations. The implementation follows the established BaseWorker pattern and integrates seamlessly with the existing BullMQ-based worker architecture.
Key changes:
- New AttachmentDeliveryWorker with concurrency of 2 for async file processing
- Bot composers refactored to queue attachment jobs instead of synchronous file delivery
- Extraction of common worker bot creation logic into
createWorkerBotutility
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/workers/attachment-delivery/worker.ts | Core worker implementation handling file downloads, media group delivery, and error handling |
| src/workers/attachment-delivery/queue.ts | Queue configuration with job interface and helper functions |
| src/workers/attachment-delivery/startup.ts | Worker startup with health checks and monitoring setup |
| src/workers/attachment-delivery/index.ts | Module exports for queue and worker |
| src/configs/attachmentDeliveryWorker.ts | Zod-validated configuration schema for worker health check port |
| src/bot/utils/createWorkerBot.ts | Extracted utility for creating bot instances used by workers |
| src/workers/broadcasts/worker.ts | Updated to use new createWorkerBot utility |
| src/workers/shared/queueRegistry.ts | Registered new attachment delivery queue for Bull Board monitoring |
| src/workers/index.ts | Added export for attachment-delivery module |
| src/bot/composers/lookups/exam-timetable/composer.ts | Refactored to queue attachment jobs instead of direct file delivery |
| src/bot/composers/lookups/announcements/composer.ts | Refactored to queue attachment jobs with batch processing support |
| src/bot/composers/lookups/academic-calendar/composer.ts | Refactored to queue attachment jobs instead of direct file delivery |
| src/bot/composers/inlineQuery/composer.ts | Refactored inline query result handler to use attachment queue |
| docker/compose.yaml | Added attachment-delivery-worker service to production compose |
| docker/compose.staging.yaml | Added attachment-delivery-worker service to staging compose |
| docker/compose.dev.yaml | Added attachment-delivery-worker service to dev compose with hot reload |
| env/dev/attachment-delivery-worker.env | Dev environment configuration for worker health check port |
| env/staging/attachment-delivery-worker.env | Staging environment configuration for worker health check port |
| env/prod/attachment-delivery-worker.env | Production environment configuration for worker health check port |
| package.json | Added npm scripts for running attachment-delivery-worker in dev and prod |
| docs/working.md | Updated architecture documentation with attachment worker details |
| README.md | Added attachment delivery worker to services table and startup commands |
| CLAUDE.md | Updated development guide with attachment worker information |
| .tool-versions | Added Node.js version specification (v20) |
- Add missing env file reference in production docker-compose.yaml - Add missing networks configuration in staging docker-compose.yaml - Remove redundant totalAttachments field from AttachmentDeliveryJob interface - Fix Zod schema validation to be consistent with other worker configurations - Revert incorrect error re-throwing (follow broadcasts worker pattern) All changes verified with TypeScript compilation and build
- Created src/workers/shared/utils/telegramErrorHandler.ts - Eliminated ~120 lines of duplicated error handling code - Reduced BroadcastsWorker from 263 to 185 lines (-78 lines) - Reduced AttachmentDeliveryWorker from 251 to 196 lines (-55 lines) - Centralized user blocked/deactivated/rate limit handling - All quality checks passed (typecheck, lint, build) No functional changes - pure refactoring for better maintainability
- Added 'ktu-bot-network' definition to docker/compose.staging.yaml - Fixes error: 'service refers to undefined network ktu-bot-network' - Network configuration now consistent with dev and prod environments
- Academic Calendar: "Downloading your calendar in the background... This may take a moment!" - Exam Timetable: "Fetching your timetable in the background... Please wait!" - Announcements: "Downloading your file(s) in the background... This may take a moment!" - Inline Query: "Downloading your file(s) from [resource] in the background... Please wait!" Changes make it clear that processing happens in the background and set realistic expectations
devadathanmb
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new Attachment Delivery Worker to the project, enabling asynchronous file downloads and delivery to users without blocking the bot. The changes span configuration, documentation, Docker setup for all environments, and bot logic refactoring to leverage the new worker. This improves scalability, user experience, and error handling for delivering attachments.
Attachment Delivery Worker Implementation
attachment-delivery-workerservice to all Docker Compose files (docker/compose.dev.yaml,docker/compose.staging.yaml,docker/compose.yaml) with appropriate environment variables, health checks, and resource limits. [1] [2] [3] [4] [5] [6] [7]env/dev/attachment-delivery-worker.env,env/staging/attachment-delivery-worker.env,env/prod/attachment-delivery-worker.env).package.jsonfor running the worker in dev and production modes..tool-versionsto specify Node.js 20 for compatibility.Bot Logic Refactoring
src/bot/composers/inlineQuery/composer.tsto queue attachment delivery jobs via the new worker instead of downloading and sending files directly in the bot process. [1] [2] [3]Documentation and Architecture Updates
README.md,CLAUDE.md, anddocs/working.mdto describe the new worker's responsibilities, architecture, and developer workflow. Added diagrams and updated service/port references. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]These changes collectively improve the system's ability to handle file attachments efficiently and robustly, while keeping the bot responsive for all users.