Background Job Queue Architecture

[rohitdash08]

Feature

Move heavy operations (imports, AI, reminders) to async job queue.

Why This Matters

Critical for scale & performance.

Acceptance Criteria

• Async processing with retries
• Dead-letter handling
• Monitoring visibility

Technical Notes

Redis + worker pattern recommended.

Test Plan

• Retry & failure handling tests

[samiyadev786]

@rohitdash08 I’d like to work on this—could you assign it to me?

[rohitdash08]

@samiyadev786 you can start

[samiyadev786]

@samiyadev786 Thanks Sir!

[rohitdash08]

@samiyadev786 welcome mam,

can I know your approach before assigning?

[samiyadev786]

@rohitdash08 Sure

[samiyadev786]

Approach for Background Job Queue Architecture

Hi @rohitdash08, here's my planned approach:

1. Job Queue Model (new table: job_queue)

• Fields: id, user_id, job_type (e.g. ai.budget_suggestion, import.pdf, reminder.send), payload (JSON), status (PENDING/RUNNING/COMPLETED/FAILED/DEAD), result (JSON), attempt (int), max_retries, next_run_at, started_at, completed_at, error_message, created_at
• This gives us persistent job tracking with full visibility into every job's lifecycle

2. Redis + Worker Pattern

• Redis (already in the stack) acts as the broker — new jobs are pushed to a Redis list (finmind:jobs queue) for instant pickup
• A dedicated worker loop (using APScheduler, already installed) polls the queue and processes jobs
• The worker runs inside a separate thread/process spawned at app startup, keeping the API responsive
• Each job type maps to a handler function — clean dispatch pattern via a JOB_HANDLERS registry dict

3. Three Job Types to Migrate

• AI Insights (services/ai.py) — budget suggestions with OpenAI calls (can take 5-10s)
• Statement Import (services/expense_import.py) — PDF/CSV parsing + Gemini AI extraction (heavy I/O)
• Reminder Delivery (services/reminders.py) — email via SMTP + WhatsApp via Twilio (external network calls)

4. Retry with Exponential Backoff

• On failure: increment attempt, compute next_run_at using exponential backoff (30s, 2min, 8min, 30min)
• Default max_retries = 5, configurable per job type
• Worker picks up retryable jobs whose next_run_at has passed

5. Dead-Letter Handling

• After max_retries exhausted, job status moves to DEAD
• Dead jobs are preserved in DB for inspection (not deleted)
• GET /admin/jobs/dead endpoint lists dead-letter jobs with error details
• POST /admin/jobs/{id}/retry allows manual re-enqueue of dead jobs

6. Monitoring Visibility

• GET /admin/jobs — list jobs with filters (status, job_type, date range)
• GET /admin/jobs/stats — summary counts by status (pending, running, completed, failed, dead) + avg processing time
• GET /admin/jobs/{id} — full detail of a single job including all attempts and error history
• Redis cache for stats dashboard with short TTL for performance

7. API Integration

• Existing routes (expense import, budget suggestion, reminder trigger) updated to enqueue jobs instead of running inline
• Returns 202 Accepted with job_id so clients can poll status
• GET /jobs/{id}/status — lightweight poll endpoint for job progress

8. Tests

• Job enqueue and dequeue lifecycle
• Retry logic: verify attempt count, backoff timing, status transitions
• Dead-letter: verify jobs move to DEAD after max retries
• Worker handler dispatch for each job type
• Concurrency: ensure no duplicate processing of same job

No new dependencies — builds entirely on Redis + APScheduler + SQLAlchemy already in the project.

Happy to adjust anything before I start!

[rohitdash08]

@samiyadev786 Thanks for the detailed approach this looks solid. let’s use Celery, RabbitMQ and Flower instead of a custom worker loop.

[BennyTheBuilder]

Hey, I'm interested in this one. I've built async job systems for heavy lifting at 15 Research Lab and Authensor, so I know the pain points here.

I can handle the queue architecture with proper retry logic and dead-letter handling. Monitoring's important too - I'd set up visibility from day one so you're not flying blind.

I work in Python and TypeScript, so whatever stack you're using should be fine. Happy to start with a design doc if that helps.

What's your timeline looking like?

[rohitdash08]

@BennyTheBuilder, you can work on this issue but requirement will be still the same rabbitmq, celery and flower

[pabloDarkmoon24]

Hi! I've worked on a fix for this issue.

PR: #631

The implementation addresses the requirements described here. Happy to iterate based on your feedback!