Skip to content

Add GitHub Copilot instructions and agent configuration for Celery/Django patterns #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danieleguido merged 13 commits into from
Jan 15, 2026
Merged

Add GitHub Copilot instructions and agent configuration for Celery/Django patterns #132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
8cb46cf
Initial plan
Copilot Jan 15, 2026
57de12d
Add copilot instructions and agent configuration files
Copilot Jan 15, 2026
7ef5489
Add documentation agent instructions
Copilot Jan 15, 2026
e310ddf
Add README for .github directory
Copilot Jan 15, 2026
018722d
Merge branch 'develop' into copilot/setup-agents-and-instructions
danieleguido Jan 15, 2026
af69589
Remove collection and export task documentation from agent files
Copilot Jan 15, 2026
fbb68c0
Update documentation agent file to remove export command examples
Copilot Jan 15, 2026
10b83d3
Update README.md
danieleguido Jan 15, 2026
8f01cb3
Remove last Solr reference from README
Copilot Jan 15, 2026
ba21360
Merge branch 'copilot/setup-agents-and-instructions' of https://githu…
danieleguido Jan 15, 2026
ac95d8c
Create AGENTS.md
danieleguido Jan 15, 2026
21bff92
Update docker-build-publish.yml
danieleguido Jan 15, 2026
2b02e1e
Update Dockerfile
danieleguido Jan 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions .github/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# GitHub Copilot Agent Configuration

This directory contains configuration files for GitHub Copilot and specialized AI agents.

## Files Overview

### Main Instructions
- **`copilot-instructions.md`** - Main instructions for GitHub Copilot with repository overview, conventions, and guidelines

### Agent-Specific Instructions (`agents/` directory)
- **`celery-tasks.md`** - Guidelines for developing and maintaining Celery background tasks
- **`django-development.md`** - Django application development patterns and best practices
- **`testing.md`** - Testing framework, patterns, and conventions
- **`documentation.md`** - Documentation standards and writing guidelines

## Purpose

These files provide:

1. **Context for AI Assistants** - Help GitHub Copilot and other AI tools understand the codebase structure and conventions
2. **Onboarding Documentation** - Guide new developers on project patterns and practices
3. **Consistency** - Ensure consistent coding style and patterns across the codebase
4. **Best Practices** - Document proven patterns for common tasks

## Usage

### For GitHub Copilot
GitHub Copilot automatically reads `.github/copilot-instructions.md` to understand project conventions.

### For Specialized Agents
Agent-specific instruction files in `.github/agents/` provide detailed guidance for:
- Celery task development with job tracking
- Django models, views, admin interface, and management commands
- Writing comprehensive tests with proper mocking and assertions
- Creating and maintaining project documentation

## Repository Overview

**impresso-user-admin** is a Django application that manages user-related information for the Impresso project. Key features:

- **Background Processing**: Celery with Redis for asynchronous tasks
- **User Management**: Django authentication with custom user plans and permissions
- **Email Notifications**: Multi-format emails (text + HTML) for user actions

## Technology Stack

- Python 3.12+ with type hints
- Django web framework
- Celery task queue with Redis
- MySQL database
- Docker for containerization
- pipenv for dependency management
- mypy for type checking

## Key Concepts

### Task Organization
- **`impresso/tasks/`** - Celery task definitions with decorators
- **`impresso/utils/tasks/`** - Helper functions used by tasks
- Job progress tracking via database and Redis
- User-based permissions

### User Permissions
- User groups for different plans (Basic, Researcher, Educational)
- UserBitmap for fine-grained access control
- Profile with user-specific settings

### Development Workflow
```bash
# Start services
docker compose up -d

# Run Django server
ENV=dev pipenv run ./manage.py runserver

# Run Celery worker (separate terminal)
ENV=dev pipenv run celery -A impresso worker -l info

# Run tests
ENV=dev pipenv run ./manage.py test

# Type checking
pipenv run mypy --config-file ./.mypy.ini impresso
```

## Contributing

When modifying these instruction files:
1. Keep examples practical and based on actual code in the repository
2. Update instructions when significant patterns or conventions change
3. Ensure consistency across all agent instruction files
4. Test that instructions are clear and actionable

## Resources

- Repository: https://github.com/impresso/impresso-user-admin
- Impresso Project: https://impresso-project.ch
- License: GNU Affero General Public License v3.0
231 changes: 231 additions & 0 deletions .github/agents/celery-tasks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,231 @@
# Agent: Celery Tasks Development

This agent specializes in developing and maintaining Celery background tasks for the impresso-user-admin Django application.

## Expertise

- Creating new Celery tasks with proper decorators and configuration
- Writing helper functions for task operations
- Implementing job progress tracking
- Managing user permissions and access control
- Error handling and retry logic
- Structured logging

## Task Development Guidelines

### Task Definition Structure

All Celery tasks should follow this pattern:

```python
from celery import shared_task
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

@shared_task(
bind=True,
autoretry_for=(Exception,),
exponential_backoff=2,
retry_kwargs={"max_retries": 5},
retry_jitter=True,
)
def task_name(self, param: type) -> return_type:
"""
Task description.

Args:
param: Description

Returns:
Description
"""
logger.info(f"[context] Starting task with param={param}")
# Implementation
```

### File Organization

- **Task definitions**: Place in `impresso/tasks/`
- Use descriptive filenames ending in `_task.py` or `_tasks.py`
- Import and use helper functions from utils

- **Helper functions**: Place in `impresso/utils/tasks/`
- Reusable logic that can be called by multiple tasks
- Database operations, API calls, data processing
- Keep helpers stateless and testable

### Job Progress Tracking

For long-running tasks, use the Job model to track progress:

```python
from impresso.models import Job
from impresso.utils.tasks import (
update_job_progress,
update_job_completed,
is_task_stopped,
TASKSTATE_PROGRESS,
)

def long_running_task(self, job_id: int):
job = Job.objects.get(pk=job_id)

# Check if user stopped the job
if is_task_stopped(task=self, job=job, progress=0.0, logger=logger):
return

# Update progress
update_job_progress(
task=self,
job=job,
progress=0.5, # 50%
taskstate=TASKSTATE_PROGRESS,
extra={"current_step": "processing"},
message="Processing data...",
logger=logger,
)

# Complete the job
update_job_completed(
task=self,
job=job,
extra={"results": "summary"},
message="Task completed successfully",
logger=logger,
)
```

### Email Operations

Use the email utility functions:

```python
from impresso.utils.tasks.email import send_templated_email_with_context
from django.conf import settings

success = send_templated_email_with_context(
template='notification_name', # Uses emails/notification_name.txt and .html
subject='Email Subject',
from_email=f"Impresso Team <{settings.DEFAULT_FROM_EMAIL}>",
to=[user.email],
cc=[settings.DEFAULT_FROM_EMAIL],
reply_to=[settings.DEFAULT_FROM_EMAIL],
context={
'user': user,
'custom_data': 'value',
},
logger=logger,
fail_silently=False,
)
```

Implement proper error handling with retries:

```python
from django.db.utils import IntegrityError
from requests.exceptions import RequestException

@shared_task(
bind=True,
autoretry_for=(RequestException, IntegrityError),
exponential_backoff=2,
retry_kwargs={"max_retries": 5},
retry_jitter=True,
)
def resilient_task(self, param: str):
try:
# Task logic
pass
except ValueError as e:
# Don't retry validation errors
logger.error(f"Validation error: {e}")
raise
except Exception as e:
# Log and let Celery handle retry
logger.exception(f"Unexpected error: {e}")
raise
```

### Logging Best Practices

Use structured logging with context:

```python
# Always include relevant IDs
logger.info(f"[job:{job.pk} user:{user.pk}] Starting operation")

# Include metrics
logger.info(
f"[job:{job.pk}] Processed {count} items in {qtime}ms "
f"(page {page}/{loops}, {progress*100:.2f}%)"
)

# Use appropriate levels
logger.debug(f"Debug info: {data}")
logger.info(f"Operation completed successfully")
logger.warning(f"Potential issue: {warning}")
logger.error(f"Error occurred: {error}")
logger.exception(f"Exception with traceback: {e}") # Includes stack trace
```

## Testing Tasks

Create tests in `impresso/tests/tasks/`:

```python
from django.test import TestCase, TransactionTestCase
from django.contrib.auth.models import User
from impresso.tasks.my_task import my_task
from django.core import mail

class TestMyTask(TransactionTestCase):
"""
Test my_task functionality.

Run with:
ENV=dev pipenv run ./manage.py test impresso.tests.tasks.TestMyTask
"""

def setUp(self):
self.user = User.objects.create_user(
username="testuser",
email="test@example.com",
password="password123"
)
# Create default groups
from impresso.signals import create_default_groups
create_default_groups(sender="impresso")

def test_task_execution(self):
# Clear mail outbox
mail.outbox = []

# Run task
result = my_task(user_id=self.user.id)

# Assertions
self.assertEqual(result, expected_value)
self.assertEqual(len(mail.outbox), 1)
```

## Configuration Settings

Key Celery settings from `settings.py`:

- `CELERY_BROKER_URL` - Redis connection for Celery
- `IMPRESSO_GROUP_USER_PLAN_*` - User plan group names
- `DEFAULT_FROM_EMAIL` - Email sender address

## Key Models

- `Job` - Tracks long-running asynchronous tasks
- `UserBitmap` - User access permissions as bitmap
- `UserChangePlanRequest` - Plan upgrade/downgrade requests
- `UserSpecialMembershipRequest` - Special membership requests
- `Profile` - User profile with uid

## References

- Celery documentation: https://docs.celeryq.dev/
- Django documentation: https://docs.djangoproject.com/
Loading