File Downloader Service

A Python microservice for downloading files from SFTP servers using Celery for asynchronous processing and Redis for status tracking.

Features

• SFTP File Download: Download files from SFTP servers with progress tracking
• Asynchronous Processing: Use Celery for background file downloads
• Status Tracking: Real-time download status via Redis
• Error Handling: Comprehensive error handling with retry mechanisms
• Configuration: Environment-based configuration for easy deployment

Architecture

The service follows the architecture described in your requirements:

• PostgreSQL: Stores server and file metadata
• Redis: Tracks file download status and progress
• Celery: Handles asynchronous file download tasks
• Paramiko: SFTP client for file transfers

Configuration

The service uses environment variables for configuration:

Variable	Default	Description
DATABASE_URL	postgresql://user:password@localhost/file_importer	PostgreSQL connection string
REDIS_HOST	localhost	Redis server hostname
REDIS_PORT	6379	Redis server port
REDIS_DB	0	Redis database number
DOWNLOAD_DIR	/tmp/file_downloads	Local directory for file downloads
MAX_FILE_SIZE_MEMORY	10485760	Max file size for memory download (10MB)
DOWNLOAD_TIMEOUT	300	Download timeout in seconds
MAX_RETRIES	3	Maximum retry attempts
RETRY_DELAY	60	Delay between retries in seconds

Database Schema

Servers Table

• id: Primary key
• hostname: SFTP server hostname
• port: SFTP port (default: 22)
• username: SFTP username
• password: SFTP password
• path: Server directory path

Files Table

• id: Primary key
• server_id: Foreign key to servers table
• filename: File name
• size: File size in bytes
• created_at: File detection timestamp
• processed_at: Processing completion timestamp

Redis Keys

• file_status:{file_id}: File download status and progress
• server_status:{server_id}: Server availability status
• server_lock:{server_id}: Server scanning lock
• lock:file:{file_id}: File processing lock
• queue:retry: Failed files for retry