Skip to content

feat: add AI-powered policy generation command#43

Open
reaandrew wants to merge 6 commits intomainfrom
feat/ai
Open

feat: add AI-powered policy generation command#43
reaandrew wants to merge 6 commits intomainfrom
feat/ai

Conversation

@reaandrew
Copy link
Owner

Summary

  • Add new generate command that creates security-focused IAM policies from AWS service authorization documentation using LLM APIs
  • Scrapes IAM actions, condition keys, and resource types automatically from AWS docs
  • Supports parallel batch processing with configurable concurrency for faster generation
  • Includes custom prompt support for specifying security requirements
  • Implements 24-hour caching of AWS documentation pages
  • Generates comprehensive output: policy JSON, IAM reference data, and human-readable documentation

Features

  • Automatic scraping of IAM data from AWS service authorization pages
  • Parallel LLM requests with configurable concurrency (default: 3)
  • Custom prompts via --prompt flag for specific security requirements
  • Retry logic with exponential backoff for API failures (5 attempts)
  • 24-hour caching in ~/.cache/politest/ to reduce API calls
  • Three output files:
    • {service}-iam-reference.json - Scraped IAM data
    • {service}-full-access-policy.json - Generated policy with security conditions
    • {service}-policy-documentation.md - Documentation explaining each statement
  • Consistent placeholder variables (${AWS::AccountId}, ${VpcEndpointId}, etc.)
  • Progress bar using schollz/progressbar library

Usage Example

./politest generate \
  --url "https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonbedrock.html" \
  --base-url "https://your-llm-api.example.com/api" \
  --model "claude-3-sonnet" \
  --api-key "$LLM_API_KEY" \
  --output "./output" \
  --concurrency 4 \
  --prompt "Require VPC endpoint conditions where supported."

Test plan

  • Build compiles successfully
  • Generate command produces all three output files
  • Parallel processing works with configurable concurrency
  • Custom prompts are incorporated into policy generation
  • Caching works for subsequent runs
  • Retry logic handles API timeouts (HTTP 5xx)
  • README documentation updated

🤖 Generated with Claude Code

reaandrew and others added 6 commits November 23, 2025 10:47
Add new `generate` command that creates security-focused IAM policies
from AWS service authorization documentation using LLM APIs.

Features:
- Scrapes IAM actions, condition keys, and resource types from AWS docs
- Uses OpenAI-compatible LLM APIs for policy generation
- Parallel batch processing with configurable concurrency
- Custom prompt support for specific security requirements
- 24-hour caching of AWS documentation pages
- Generates three output files:
  - {service}-iam-reference.json (scraped data)
  - {service}-full-access-policy.json (generated policy)
  - {service}-policy-documentation.md (human-readable docs)
- Consistent placeholder variables (${AWS::AccountId}, ${VpcEndpointId}, etc.)
- Retry logic with exponential backoff for API failures
- Progress bar using schollz/progressbar library

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Remove getSecurityPolicySystemPrompt and buildPolicyGenerationPrompt
functions that were left over from earlier iterations before switching
to batch processing.

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add --scp flag to generate companion Service Control Policy
- SCP uses enterprise NotAction allowlist pattern
- Move security guardrails to SCP (SecureTransport, region locks, CRIS)
- Identity policy now contains only allows (actions + resources)
- Add AI-powered statement deduplication and consolidation
- Add examples.sh with Bedrock invoke-only policy example
- Update .gitignore for generated policy files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Extract duplicated validFields maps to package-level variables
  (validIAMTopLevelFields, validIAMStatementFields) in policy.go
- Add comprehensive tests for llm.go (LLM client, policy generation)
- Add tests for scraper.go (HTML parsing, cache functions)
- Add tests for generate.go (config validation)
- Add tests for progress.go (progress reporting)

Coverage improvement: internal package now at 80.1% (was ~0.25% for new code)
Duplication reduction: removed 3 instances of duplicated validation maps

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add additional tests for:
- LLM error cases and edge cases (ConsolidateStatementGroup, GeneratePolicyDocumentation, GenerateSCP, EnrichActionDescriptions)
- Mock progress reporter for testing
- Cache functions (loadFromCache, saveToCache) edge cases
- Dedupe and grouping edge cases

Coverage improved from 71.1% to 84.0% on new code, exceeding 80% threshold.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive tests for:
- ConsolidateStatementGroup with markdown fences, API errors
- GenerateSecurityPolicy with progress, default concurrency, empty access level
- ChatCompletion error cases (HTTP errors, invalid JSON, empty choices)
- doRequest with/without API key
- EnrichActionDescriptions with progress and many actions
- loadFromCache with valid, expired, and invalid JSON entries
- saveToCache with actual cache file verification
- parseIAMDocumentation with progress reporter
- extractActions, extractConditionKeys, extractResourceTypes edge cases

Key improvements:
- loadFromCache: 46.2% → 92.3%
- ConsolidateStatementGroup: 70.8% → 100%
- GenerateSecurityPolicy: 79.2% → 98.6%
- parseIAMDocumentation: 78.9% → 94.7%

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jan 6, 2026

Quality Gate Failed Quality Gate failed

Failed conditions
79.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant