Add Claude workflows for auto-labeling issues and PR creation #146
Conversation
|
Warning Rate limit exceeded@Sachindu-Nethmin has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 10 minutes and 48 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughAdds a repository-level Claude system prompt and two GitHub Actions workflows: one to auto-label issues via Claude classification, and one to locate queued issues, rotate labels (Queued → In-Progress), post trigger comments, and invoke Claude to produce PRs for documentation fixes. Changes
Sequence Diagram(s)sequenceDiagram
participant GH as GitHub Issues
participant AutoLabel as auto_label.yml
participant ClaudeAPI as Claude API
participant Labels as Issue Labels
GH->>AutoLabel: Issue OPENED/EDITED
AutoLabel->>AutoLabel: Check assignees & excluded labels
alt Skip (assigned or excluded)
AutoLabel->>GH: Exit
else Classify
AutoLabel->>ClaudeAPI: Send classification prompt
ClaudeAPI-->>AutoLabel: Return category
alt Valid category
AutoLabel->>Labels: Apply AI-Agent/Queued
Labels-->>GH: Label added
else No category
AutoLabel->>GH: No label applied
end
end
sequenceDiagram
participant GH as GitHub
participant Finder as get_issue_and_assign job
participant Runner as run_claude job
participant Repo as Repository
participant ClaudeAction as Claude Code Action
participant Labels as Issue Labels
GH->>Finder: Trigger (comment / schedule / manual)
Finder->>GH: Locate oldest AI-Agent/Queued issue
alt Issue found
Finder->>Labels: Remove Queued, Add In-Progress
Finder->>GH: Post trigger comment
GH->>Runner: Runner triggered by comment
Runner->>Repo: Checkout repository (full history)
Runner->>Runner: Prepare system prompt (envsubst), set ISSUE_NUMBER/REPOSITORY
Runner->>ClaudeAction: Invoke Claude (model, tools, args)
ClaudeAction->>Repo: Create PR / push changes
Runner->>GH: Fetch next queued issue
alt Next exists
Runner->>Labels: Update labels & post comment
end
else No queued issue
Finder->>GH: Exit gracefully
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.github/workflows/auto_label.yml:
- Around line 46-77: The PROMPT variable is built but not passed into the API
payload: update the DATA construction to use the actual PROMPT variable instead
of the hardcoded truncated string by adding an --arg prompt "$PROMPT" to the jq
call and replacing the messages content concatenation that currently uses "You
are a GitHub issue classifier for documentation fixes..." with the $prompt
value; specifically modify the DATA assignment so messages: [{role: "user",
content: $prompt}] (or equivalent jq string concatenation using $prompt) and
remove the old hardcoded text so Claude receives the full instructions.
🧹 Nitpick comments (2)
.github/workflows/auto_label.yml (1)
36-42: Potential issue with Hacktoberfest label detection when no labels exist.If the issue has no labels,
jq -r '.labels[].name'outputs nothing, andgrep -iexits with status 1, which could cause issues depending on shell settings. Consider usinggrep -cor adding|| true.♻️ Proposed fix for robust label check
# Check if issue is part of Hacktoberfest project - HAS_HACKTOBERFEST_LABEL=$(gh issue view "$ISSUE_NUMBER" --repo ${{ github.repository }} --json labels | jq -r '.labels[].name' | grep -i '^hacktoberfest$' | wc -l) + HAS_HACKTOBERFEST_LABEL=$(gh issue view "$ISSUE_NUMBER" --repo ${{ github.repository }} --json labels | jq -r '.labels[].name' | grep -ci '^hacktoberfest$' || echo "0").github/workflows/claude_runner.yml (1)
134-138: Note:exit 0in a step doesn't skip subsequent steps in GitHub Actions.The
exit 0provides a clean log message but the subsequent step (lines 140-152) will still be evaluated. This works correctly because of theifcondition on line 143, but the "Exit if no queued issues" step name might be slightly misleading since it doesn't actually exit the job.Consider renaming to "Log when no queued issues" for clarity, or convert to a single step that conditionally executes the trigger logic.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
.github/claude/system_prompt.txt.github/workflows/auto_label.yml.github/workflows/claude_runner.yml
🧰 Additional context used
🪛 LanguageTool
.github/claude/system_prompt.txt
[style] ~70-~70: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...ssue is related to by reading the issue very carefully.Then according to the issue type,apply ...
(EN_WEAK_ADJECTIVE)
[uncategorized] ~90-~90: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...n state or cookie requirements * Rate limiting or temporary access restrictions -...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🔇 Additional comments (6)
.github/claude/system_prompt.txt (1)
1-233: Well-structured system prompt for the documentation fixer.The prompt comprehensively covers issue handling, branch/PR procedures, Microsoft Style Guide compliance, and error handling. The structure is clear and the rules are well-defined.
A few minor observations:
- Line 70: Consider "reading the issue thoroughly" instead of "very carefully" for better style consistency with the Microsoft Style Guide you're enforcing.
- The prompt correctly enforces single-branch workflow and includes appropriate safeguards for image handling and external link verification.
.github/workflows/auto_label.yml (1)
79-109: Good error handling for API responses.The HTTP status capture, response body handling, and category extraction with fallback to empty string are well implemented. The validation against empty, null, and "none" values before labeling is appropriate.
.github/workflows/claude_runner.yml (4)
54-59: Job condition requires issue_comment context, but this job also handles chained triggers.The
run_claudejob is conditioned ongithub.event.comment.bodycontaining the trigger phrase, which is correct for issue_comment triggers. However, be aware this job only runs on issue_comment events and won't run for schedule/workflow_dispatch (which is handled byget_issue_and_assign).
78-104: Good handling of multi-line environment variable with unique delimiter.The use of
envsubstfor template substitution and a unique delimiter (with timestamp and PID) to safely handle multi-line content in$GITHUB_ENVis well implemented.
120-152: Potential recursive workflow triggering may have unintended consequences.After
run_claudecompletes, it fetches the next queued issue and posts a@wso2-engineering-botcomment (line 152). This triggers anotherrun_claudejob immediately, creating a chain. While the concurrency group prevents parallel execution, consider:
- Rate limits: Multiple consecutive runs might hit GitHub API or Anthropic rate limits.
- Workflow run limits: GitHub has limits on workflow run queuing.
The hourly schedule already processes queued issues, so this chaining may be redundant or could cause issues during high-volume periods.
Is the continuous chaining intentional for faster processing, or would relying solely on the hourly schedule be sufficient?
116-118: Allowed tools configuration is appropriately scoped for the use case.The tool permissions include necessary capabilities for git operations, file editing, and GitHub API interactions. The
Bash(*)wildcard is permissive but required for themkdocs build --strictverification mentioned in the system prompt.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In @.github/claude/system_prompt.txt:
- Line 68: Replace the run-on "carefully.Then" with proper spacing and clearer
phrasing: change the sentence to something concise like "First, read the issue
carefully. Then, based on the issue type, apply the appropriate solution."
Ensure you add the missing space after the period and prefer "read" over
"understand" for brevity.
🧹 Nitpick comments (5)
.github/claude/system_prompt.txt (5)
35-59: Clarify branching and PR workflow language.The instructions mix "working on the main branch" (line 36) and "create a single PR targeting the main branch" (lines 37, 56–58). This is technically correct—you check out a branch from main, commit to it, then open a PR back to main—but the phrasing could confuse Claude about which branch to work on. The step-by-step checkout and git commands (lines 42–44) clarify intent, but the prose is ambiguous.
♻️ Proposed clarification
Replace lines 35–37 with:
3. Branch selection: - - Always work on the main branch. - - Create a single PR targeting the main branch. + - Create a new feature branch from main (see step 4 below). + - Always target main as the base branch for the PR.This removes the ambiguity that you "work on main" when you actually work on a feature branch checked out from main.
81-91: External link verification guidance needs practical constraints.Lines 81–91 provide detailed guidance on handling external link verification, but the instructions are open-ended and subjective. Phrases like "multiple verification methods," "try multiple verification approaches," and "likely valuable" lack concrete thresholds. This risks inconsistent behavior or Claude attempting excessive manual verification.
♻️ Proposed refinement
Tighten the guidance with concrete criteria:
- - If a link appears to have access restrictions but contains relevant WSO2 API Manager content, consider it valid - - External sites (Medium, blogs, forums, documentation portals) may return different responses based on: + - External sites may return different responses based on: * Geographic region, network conditions, or request headers * Authentication state or cookie requirements * Rate limiting or temporary access restrictions - - For important technical resources, extract and include key information in the PR to demonstrate content validity - - When in doubt, include the link with appropriate context rather than removing potentially valuable references - - Check for alternative official documentation before discarding external referencesAnd add a clear fallback:
+ - **Default approach**: When a link returns a temporary/auth error but appears to be an official resource (WSO2 docs, official blogs): + * Include the link in the PR with a comment noting the verification challenge + * Do not attempt exhaustive alternative verification methods + - Only mark a link as invalid if it returns a permanent 404/410 or contains clearly irrelevant contentThis sets clear boundaries on verification effort and reduces ambiguity.
110-120: Image verification logic could be more actionable.Lines 110–116 instruct to "verify that the image exists in the repository" and "Only use images that are confirmed to exist," but the method to check is not specified. This may be difficult for Claude to execute reliably. Similarly, lines 119–120 prioritize issue types but the context is unclear without re-reading earlier sections.
♻️ Proposed clarification
Add a specific verification method:
**Step 5: Image Handling for New Documents** - If adding or modifying image references: - First verify the image exists in the repository + * Check the repository directory structure under `docs/` or the image asset path referenced in the file + * Use exact relative paths as shown in other documentation files - Check if the image is accessible and properly displays in the documentation - Only add references to images that are confirmed to exist in the repository - If needed images don't exist, note this in PR comments but don't add broken image links - Use relative paths to reference images following repository conventionsAlso, move the priority list (lines 118–119) into a separate, clearly labeled section or place it immediately after the issue type definitions for easier reference:
+ +**Issue Resolution Priority**: +When an issue spans multiple categories, solve in this order: +Broken Links > Spelling > Grammar > Documentation > Suggestions -If that issue is related to multiple issue cases then use the below priority list to solve them. -Broken Links > Spelling > Grammar > Documentation > Suggestions
122-160: Consolidate redundant Microsoft Style Guide statements.The Microsoft Style Guide compliance requirement is stated multiple times:
- Line 18 (new documents must 100% adhere)
- Line 19 (editing existing: apply only to new content)
- Lines 122–125 (repeat the requirement verbatim)
- Lines 149–154 (new documents: full compliance)
- Lines 156–160 (existing documents: style only for new content)
This redundancy risks Claude prioritizing conflicting statements or overthinking the rule. The core logic is sound, but the presentation should be consolidated.
♻️ Proposed consolidation
Replace lines 122–160 with a single, structured section:
========================================= DOCUMENTATION STYLE GUIDELINES ========================================= -All documentation changes, regardless of the issue type, MUST comply with the Microsoft Style Guide (https://learn.microsoft.com/en-us/style-guide/welcome/). - -Key rules that MUST be enforced: +All documentation changes must align with the Microsoft Style Guide (https://learn.microsoft.com/en-us/style-guide/welcome/). + +**For NEW documents**: Every aspect (structure, headings, voice, terminology, formatting, examples) must fully comply with Microsoft Style Guide. +**For EXISTING documents**: Apply Microsoft Style Guide standards ONLY to newly added content; do not modify existing content style unless the issue specifically requests formatting/style fixes. + +Key formatting rules:This removes duplication and clearly delineates the two scenarios (new vs. edit).
162-178: PR body template should clarify optional sections.The PR template (lines 167–176) includes conditional sections ("Include ONLY when...") but the syntax is embedded in prose. This may cause Claude to include all sections or misinterpret conditions.
💡 Proposed improvement
Reformat the template to make conditions explicit and easier to parse:
- PR body template: + Use the template below, including ONLY the sections marked [WHEN APPLICABLE]: This PR was automatically generated by Claude AI. - Issue: LINK OF THE ISSUE - Type: [Broken Links / Spelling / Grammar / Documentation / Suggestions] - Summary: [1–2 line description of changes] + + [WHEN APPLICABLE: Creating new documentation] - New Document Verification: [Include ONLY when creating new documentation] CONFIRM that this new document FULLY COMPLIES with ALL Microsoft Style Guide requirements. Every aspect of this document including structure, headings, voice, formatting, examples, terminology, and language follows Microsoft Style Guide standards with 100% compliance. + + [WHEN APPLICABLE: Adding to existing documents] - Style Scope Verification: [Include ONLY when adding to existing documents] Verify Microsoft Style Guidelines have been applied ONLY to newly added content without modifying existing content style unless specifically requested. + + [WHEN APPLICABLE: References to images] - Image Verification: [Include ONLY when creating new documentation] Verify that all referenced images exist in the repository and are accessible. No broken image links have been added. + - Verification: mkdocs build --strict passedThis makes it harder for Claude to accidentally include irrelevant sections.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
.github/claude/system_prompt.txt.github/workflows/claude_runner.yml
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/claude_runner.yml
🧰 Additional context used
🪛 LanguageTool
.github/claude/system_prompt.txt
[style] ~68-~68: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...ssue is related to by reading the issue very carefully.Then according to the issue type,apply ...
(EN_WEAK_ADJECTIVE)
[uncategorized] ~88-~88: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...n state or cookie requirements * Rate limiting or temporary access restrictions -...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
🔇 Additional comments (1)
.github/claude/system_prompt.txt (1)
1-231: Request verification of placeholder substitution mechanism.The prompt uses template variables like
${REPOSITORY},${ISSUE_NUMBER},${BRANCH_NAME}, and${TIMESTAMP}throughout. It's unclear whether these are:
- Meant to be substituted by the calling workflow (claude_runner.yml) before passing to Claude
- Meant to be substituted by Claude during execution
- Expected to remain as shell variables for Claude to reference in git commands
If these variables are not pre-populated by the workflow, Claude may output literal
${VAR}strings in git commands or PR descriptions, causing failures.Please verify:
- Does
claude_runner.ymlsubstitute these variables before invoking Claude's API?- If so, provide examples of the substituted prompt for a sample issue.
- If Claude is expected to substitute them, clarify in the prompt that these are template variables Claude must interpolate at runtime.
1 participant
Purpose
Enable automated documentation issue fixing for the Devant documentation repository. Addresses the need for faster resolution of common documentation issues (broken links, spelling, grammar, formatting) directly on the main branch.
Goals
Approach
Added two configuration files
1. Claude Runner Workflow (
.github/workflows/claude_runner.yml):@wso2-engineering-botmentions in issue commentsAI-Agent/Queuedlabelmainbranchmkdocs build --strict2. System Prompt (
.github/claude/system_prompt.txt):fixing-issue-{ISSUE_NUMBER}-{TIMESTAMP}Summary by CodeRabbit
Documentation
Chores
✏️ Tip: You can customize this high-level summary in your review settings.