Skip to content

Conversation

@jfrench9
Copy link
Member

Summary

This PR migrates the robosystems client from Celery-based background task processing to Dagster background tasks for handling long-running operations. This change modernizes our async operation infrastructure and improves integration with our data pipeline orchestration.

Key Changes

  • Background Task Migration: Replaced Celery task handling with Dagster background task patterns across agent operations and file processing
  • Client Architecture Updates: Refactored materialization client with enhanced async operation handling (+92/-56 lines)
  • SSE Client Improvements: Streamlined server-sent events client implementation for better real-time operation monitoring
  • Model Simplification: Cleaned up materialize response models by removing Celery-specific attributes and dependencies
  • Configuration Cleanup: Removed Celery-related configuration from gitignore and pytest settings

Breaking Changes

⚠️ This is a breaking change for any consumers relying on:

  • Celery task IDs in materialization responses
  • Celery-specific error handling patterns
  • Previous async operation callback structures

Files Modified

  • API Endpoints (3 files): Updated agent execution and file update endpoints for new async patterns
  • Client Extensions (4 files): Enhanced materialization, SSE, and workspace clients with Dagster integration
  • Models (1 file): Simplified response structures to remove Celery dependencies
  • Configuration (2 files): Removed legacy Celery references

Testing Notes

  • Verify all async operations (agent execution, file updates) complete successfully
  • Test materialization workflows end-to-end with new background task handling
  • Confirm SSE connections properly stream operation status updates
  • Validate error handling and timeout scenarios with Dagster tasks

Infrastructure Considerations

  • Dagster background task infrastructure must be properly configured before deployment
  • Monitor task execution patterns and performance compared to previous Celery implementation
  • Ensure proper cleanup of completed background tasks to prevent resource accumulation
  • Update monitoring and alerting to track Dagster task metrics instead of Celery queues

🤖 Generated with Claude Code

Branch Info:

  • Source: feature/rename-dagster-background-tasks
  • Target: main
  • Type: feature

Co-Authored-By: Claude noreply@anthropic.com

- Updated documentation to replace references to "Async Celery worker" with "Background queue" for long operations in `auto_select_agent.py`, `execute_specific_agent.py`, and `update_file.py`.
- Enhanced the `MaterializeResponse` model to include an operation ID for better tracking of materialization jobs.
- Improved SSE client handling by introducing context managers for better resource management during streaming operations.
- Adjusted timeout settings for SSE streaming in `subgraph_workspace_client.py` to accommodate longer-running jobs.

These changes improve clarity in operation handling and enhance the robustness of the streaming and materialization processes.
- Removed emoji prefixes from section titles for a cleaner presentation.
- Standardized section headings by removing unnecessary symbols.
- Updated print statements to improve clarity in examples and troubleshooting sections.

These changes aim to improve the readability and professionalism of the documentation.
@jfrench9 jfrench9 merged commit b85f27f into main Dec 22, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants