⚡ Bolt: Parallelize rule batch pushing #90
Conversation
- Introduced `MAX_PUSH_WORKERS` (4) to process rule batches in parallel. - Refactored `push_rules` to use `ThreadPoolExecutor`. - Ensured thread safety for `existing_rules` updates. - Reduced simulated push time from ~1.0s to ~0.3s for 10 batches.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return True | ||
| except httpx.HTTPError as e: | ||
| log.error(f"Failed to push batch {i} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") | ||
| log.error(f"Failed to push batch {batch_idx} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") |
Check warning
Code scanning / Pylint (reported by Codacy)
Line too long (124/100) Warning
| return True | ||
| except httpx.HTTPError as e: | ||
| log.error(f"Failed to push batch {i} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") | ||
| log.error(f"Failed to push batch {batch_idx} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") |
Check warning
Code scanning / Prospector (reported by Codacy)
Use lazy % formatting in logging functions (logging-fstring-interpolation) Warning
| return True | ||
| except httpx.HTTPError as e: | ||
| log.error(f"Failed to push batch {i} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") | ||
| log.error(f"Failed to push batch {batch_idx} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") |
Check warning
Code scanning / Pylintpython3 (reported by Codacy)
Line too long (124/100) Warning
| return True | ||
| except httpx.HTTPError as e: | ||
| log.error(f"Failed to push batch {i} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") | ||
| log.error(f"Failed to push batch {batch_idx} for folder {sanitize_for_log(folder_name)}: {sanitize_for_log(e)}") |
Check notice
Code scanning / Pylintpython3 (reported by Codacy)
Use lazy % formatting in logging functions Note
⚡ Bolt: Parallelize rule batch pushing
💡 What:
Refactored the
push_rulesfunction to send batches of rules to the Control D API in parallel usingThreadPoolExecutor.🎯 Why:
Previously, batches were sent serially. For large folders with many rules, this resulted in long wait times (N batches * Latency). By parallelizing, we reduce the total time significantly while keeping concurrency low (4 workers) to respect API stability.
📊 Impact:
🔬 Measurement:
push_rulesstill correctly adds all rules.PR created automatically by Jules for task 1057747744472600381 started by @abhimehro