Data Carousel: submit iDDS when reusing DDM rule, distinguish staging vs staged rules, add rescue pending tasks

[mightqxc]

No description provided.

[Copilot]

Pull request overview

This PR enhances the Data Carousel system to better handle datasets that reuse existing DDM (Distributed Data Management) rules by:

• Distinguishing between datasets that are still staging vs. already staged when reusing DDM rules
• Submitting iDDS (intelligent Data Delivery Service) requests for tasks that reuse existing DDM rules
• Adding a rescue mechanism for pending tasks with already-completed data carousel requests

Key Changes:

• Split to_reuse_ds_list into to_reuse_staging_ds_list and to_reuse_staged_ds_list to differentiate staging status
• Added submit_idds_request parameter to control iDDS request submission when reusing DDM rules
• Implemented rescue_pending_tasks_with_done_requests() method to update staged files for stuck pending tasks

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pandaserver/taskbuffer/DataCarousel.py	Refactored dataset list handling to distinguish staging vs staged rules; added iDDS submission for reused rules; added rescue method for pending tasks; updated _update_staged_files to accept task ID list parameter
pandajedi/jediorder/TaskRefiner.py	Updated to use new dataset list variables; properly categorizes already-staged datasets as not needing staging vs still-staging datasets
pandajedi/jedidog/AtlasDataCarouselWatchDog.py	Added doRescuePendingTasks method with lock time constant; method is currently commented out in main action loop (staged rollout)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.