diff --git a/.gitignore b/.gitignore index ad85e06..f932d75 100644 --- a/.gitignore +++ b/.gitignore @@ -22,10 +22,25 @@ yarn-error.log* frontend/dist/ frontend/.vite/ -# Environment variables +# Environment variables and secrets .env .env.local .env.*.local +.env.production +.env.development +.env.test +*.env +*_secrets.json +*_credentials.json +secrets/ +credentials/ +*.key +*.pem +*.p12 +*.crt +*.cer +config/secrets.yml +config/credentials.yml # IDE .vscode/ diff --git a/DEPLOYMENT_GUIDE.md b/DEPLOYMENT_GUIDE.md index 862e921..dfc6bf4 100644 --- a/DEPLOYMENT_GUIDE.md +++ b/DEPLOYMENT_GUIDE.md @@ -33,7 +33,7 @@ Complete guide for deploying the PDDL RLHF system to Vercel (frontend) and Railw - Click "Variables" tab - Add variable: - Name: `FIREWORKS_API_KEY` - - Value: `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` (or your API key) + - Value: Your Fireworks AI API key (get from https://fireworks.ai/) 3. **Deployment Settings:** - Railway will auto-detect Python diff --git a/QUICK_REFERENCE.md b/QUICK_REFERENCE.md index 6361f30..1af6a89 100644 --- a/QUICK_REFERENCE.md +++ b/QUICK_REFERENCE.md @@ -64,7 +64,7 @@ npm install ### Backend ```bash -FIREWORKS_API_KEY=fw_3ZHFp8ZR5WeoadXcFcjEKY4z +FIREWORKS_API_KEY=your_fireworks_api_key_here PORT=8000 ``` diff --git a/SECURITY_MIGRATION.md b/SECURITY_MIGRATION.md new file mode 100644 index 0000000..dd3eb37 --- /dev/null +++ b/SECURITY_MIGRATION.md @@ -0,0 +1,375 @@ +# ๐Ÿ”’ Security Migration Guide - API Key Remediation + +## โš ๏ธ CRITICAL: Immediate Actions Required + +This document outlines the **mandatory steps** to complete the security remediation after removing hardcoded API keys from the codebase. + +--- + +## ๐Ÿšจ Phase 1: IMMEDIATE - Revoke Exposed API Keys + +### Exposed API Keys (MUST BE REVOKED IMMEDIATELY) + +The following Fireworks AI API keys were exposed in the git repository and **MUST** be revoked: + +1. `fw_3ZNkrZnbfKVHhU65bFirkpJr` (found in `pddl_planner.py`) +2. `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` (found in multiple documentation files) + +### Steps to Revoke Keys + +1. **Log in to Fireworks AI Dashboard:** + - Visit: https://fireworks.ai/ + - Navigate to API Keys section + +2. **Revoke Each Exposed Key:** + - Find each key listed above + - Click "Revoke" or "Delete" + - Confirm revocation + +3. **Verify Revocation:** + - Test that the old keys no longer work: + ```bash + curl -X POST https://api.fireworks.ai/inference/v1/chat/completions \ + -H "Authorization: Bearer fw_3ZHFp8ZR5WeoadXcFcjEKY4z" \ + -H "Content-Type: application/json" \ + -d '{"model": "test", "messages": [{"role": "user", "content": "test"}]}' + ``` + - Expected: Authentication error (401 Unauthorized) + +--- + +## ๐Ÿ”‘ Phase 2: Generate New API Keys + +### Create New Keys with Proper Scoping + +1. **Generate New Fireworks AI Key:** + - In Fireworks AI Dashboard, create a new API key + - Apply principle of least privilege (minimal required permissions) + - Add descriptive name: `PDDL-RLHF-Production-YYYY-MM-DD` + - Document key creation date and purpose + +2. **Store Keys Securely:** + - **DO NOT** commit to git + - Use environment variables + - For production: Use Railway/Vercel environment variable settings + - For local development: Use `.env` file (already in `.gitignore`) + +3. **Set Environment Variables:** + + **Local Development (.env file):** + ```bash + # Create .env file in project root (backend folder for backend) + echo "FIREWORKS_API_KEY=your_new_api_key_here" > backend/.env + ``` + + **Railway (Production Backend):** + ``` + 1. Go to Railway Dashboard + 2. Select your project + 3. Click "Variables" tab + 4. Add/Update: FIREWORKS_API_KEY=your_new_api_key_here + 5. Save (triggers auto-redeploy) + ``` + + **Vercel (Production Frontend - if needed):** + ``` + 1. Go to Vercel Dashboard + 2. Select your project + 3. Settings โ†’ Environment Variables + 4. Add any required API keys + 5. Redeploy + ``` + +--- + +## ๐Ÿงน Phase 3: Git History Scrubbing + +### โš ๏ธ WARNING: History Rewriting Operation + +Git history scrubbing **rewrites repository history** and requires all team members to re-clone the repository. + +### Prerequisites + +- Backup repository before proceeding +- Notify all team members +- Schedule during low-activity period +- Ensure you have force-push permissions + +### Option A: Using git-filter-repo (Recommended) + +1. **Install git-filter-repo:** + ```bash + # macOS + brew install git-filter-repo + + # Ubuntu/Debian + apt-get install git-filter-repo + + # pip + pip install git-filter-repo + ``` + +2. **Create Fresh Clone:** + ```bash + # Work on a fresh clone to avoid issues + git clone https://github.com/your-username/your-repo.git repo-cleanup + cd repo-cleanup + ``` + +3. **Create Pattern File:** + ```bash + cat > api-keys.txt << 'EOF' + fw_3ZNkrZnbfKVHhU65bFirkpJr + fw_3ZHFp8ZR5WeoadXcFcjEKY4z + EOF + ``` + +4. **Run git-filter-repo:** + ```bash + git filter-repo --replace-text api-keys.txt --force + ``` + +5. **Verify Scrubbing:** + ```bash + # Search all history for the old keys + git log --all -S "fw_3ZNkrZnbfKVHhU65bFirkpJr" + git log --all -S "fw_3ZHFp8ZR5WeoadXcFcjEKY4z" + # Should return no results + ``` + +6. **Force Push to Remote:** + ```bash + git remote add origin https://github.com/your-username/your-repo.git + git push origin --force --all + git push origin --force --tags + ``` + +### Option B: Using BFG Repo-Cleaner (Alternative) + +1. **Download BFG:** + ```bash + # Download from: https://rtyley.github.io/bfg-repo-cleaner/ + # Or use homebrew + brew install bfg + ``` + +2. **Create Fresh Mirror Clone:** + ```bash + git clone --mirror https://github.com/your-username/your-repo.git + cd your-repo.git + ``` + +3. **Create Replacement File:** + ```bash + cat > replacements.txt << 'EOF' + fw_3ZNkrZnbfKVHhU65bFirkpJr==>REDACTED_API_KEY + fw_3ZHFp8ZR5WeoadXcFcjEKY4z==>REDACTED_API_KEY + EOF + ``` + +4. **Run BFG:** + ```bash + bfg --replace-text replacements.txt + ``` + +5. **Clean and Push:** + ```bash + git reflog expire --expire=now --all + git gc --prune=now --aggressive + git push --force + ``` + +--- + +## ๐Ÿ“‹ Phase 4: Post-Migration Verification + +### Verification Checklist + +- [ ] **Old API keys revoked and non-functional** + ```bash + # Test should fail with 401 + curl -H "Authorization: Bearer fw_3ZHFp8ZR5WeoadXcFcjEKY4z" \ + https://api.fireworks.ai/inference/v1/chat/completions + ``` + +- [ ] **No hardcoded keys in current codebase** + ```bash + # Search for API key patterns + grep -r "fw_[a-zA-Z0-9]" . --exclude-dir=.git --exclude-dir=node_modules + # Should only return placeholders or this document + ``` + +- [ ] **Git history clean** + ```bash + # Search entire history + git log --all -S "fw_3ZNkrZnbfKVHhU65bFirkpJr" + git log --all -S "fw_3ZHFp8ZR5WeoadXcFcjEKY4z" + # Should return no results + ``` + +- [ ] **New API keys configured in environments** + - Railway: Variables tab shows FIREWORKS_API_KEY (value hidden) + - Local: `.env` file exists with new key (not committed) + +- [ ] **Application functional with new keys** + ```bash + # Test local backend + cd backend + python main.py + # Should start without errors + + # Test API call + curl -X POST http://localhost:8000/api/generate-plan \ + -H "Content-Type: application/json" \ + -d '{"prompt": "test", "temperature": 0.5, "max_tokens": 100}' + # Should return valid response + ``` + +- [ ] **.gitignore properly configured** + ```bash + # Verify .gitignore includes credential patterns + cat .gitignore | grep -E "(\.env|secret|credential|\.key)" + ``` + +--- + +## ๐Ÿ‘ฅ Phase 5: Team Communication + +### Required Team Actions + +**All team members MUST:** + +1. **Delete their local repository:** + ```bash + cd .. + rm -rf your-repo + ``` + +2. **Re-clone from scratch:** + ```bash + git clone https://github.com/your-username/your-repo.git + cd your-repo + ``` + +3. **Set up new API keys:** + ```bash + # Backend + cd backend + cp .env.example .env # If exists + # Edit .env and add new FIREWORKS_API_KEY + ``` + +4. **Verify setup:** + ```bash + # Test that application works with new keys + cd backend + python main.py + ``` + +### Communication Template + +``` +๐Ÿ”’ SECURITY UPDATE - ACTION REQUIRED + +We have removed exposed API keys from our repository and cleaned the git history. + +REQUIRED ACTIONS (URGENT): +1. Delete your local repository clone +2. Re-clone from GitHub: git clone +3. Request new API keys from [team lead/security contact] +4. Set up .env file with new keys (see SECURITY_MIGRATION.md) +5. DO NOT use old API keys - they have been revoked + +DEADLINE: [Set appropriate deadline] + +If you have questions, contact [security contact] +``` + +--- + +## ๐Ÿ”„ Phase 6: Ongoing Security Practices + +### Prevention Measures + +1. **Pre-commit Hooks:** + ```bash + # Install git-secrets + git secrets --install + git secrets --register-aws + + # Add custom patterns + git secrets --add 'fw_[a-zA-Z0-9]{20,}' + git secrets --add 'FIREWORKS_API_KEY=[^\\s]+' + ``` + +2. **Regular Audits:** + - Weekly: Scan codebase for hardcoded secrets + - Monthly: Rotate API keys + - Quarterly: Review access permissions + +3. **Secret Scanning:** + - Enable GitHub secret scanning (if using GitHub) + - Use tools like `truffleHog` or `detect-secrets` + +4. **Documentation:** + - Keep this document updated + - Document all API key rotations + - Maintain audit trail + +--- + +## ๐Ÿ“ž Support & Resources + +### Tools +- **git-filter-repo**: https://github.com/newren/git-filter-repo +- **BFG Repo-Cleaner**: https://rtyley.github.io/bfg-repo-cleaner/ +- **git-secrets**: https://github.com/awslabs/git-secrets +- **truffleHog**: https://github.com/trufflesecurity/truffleHog + +### Fireworks AI +- **Dashboard**: https://fireworks.ai/ +- **API Documentation**: https://docs.fireworks.ai/ +- **Support**: support@fireworks.ai + +### Emergency Contact +- For security incidents: [Add security contact] +- For technical issues: [Add technical contact] + +--- + +## โœ… Completion Checklist + +Mark each item as you complete it: + +- [ ] Phase 1: Revoked all exposed API keys +- [ ] Phase 2: Generated and configured new API keys +- [ ] Phase 3: Scrubbed git history using git-filter-repo or BFG +- [ ] Phase 4: Verified all checks pass +- [ ] Phase 5: Notified team and coordinated re-cloning +- [ ] Phase 6: Implemented prevention measures + +**Date Completed:** _______________ + +**Completed By:** _______________ + +**Verified By:** _______________ + +--- + +## ๐Ÿ“ Audit Log + +Document all key-related actions: + +| Date | Action | Performed By | Notes | +|------|--------|--------------|-------| +| YYYY-MM-DD | Keys exposed in git | - | Initial discovery | +| YYYY-MM-DD | Hardcoded keys removed | - | Code changes committed | +| | Old keys revoked | | Awaiting action | +| | New keys generated | | Awaiting action | +| | Git history scrubbed | | Awaiting action | +| | Team notified | | Awaiting action | + +--- + +**Last Updated:** [Current Date] +**Document Version:** 1.0 diff --git a/SECURITY_REMEDIATION_SUMMARY.md b/SECURITY_REMEDIATION_SUMMARY.md new file mode 100644 index 0000000..e77c20f --- /dev/null +++ b/SECURITY_REMEDIATION_SUMMARY.md @@ -0,0 +1,274 @@ +# ๐Ÿ”’ Security Remediation Summary + +**Date:** [Current Date] +**Status:** โœ… Code Changes Complete | โณ Manual Actions Required + +--- + +## โœ… Completed Actions + +### 1. Hardcoded API Keys Removed + +All hardcoded Fireworks AI API keys have been removed from the following files: + +| File | Line | Previous Value | Current Implementation | +|------|------|----------------|------------------------| +| `pddl_planner.py` | 14 | `fw_3ZNkrZnbfKVHhU65bFirkpJr` | `os.getenv("FIREWORKS_API_KEY")` with validation | +| `test_pddl_direct.sh` | 9 | `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` | `$FIREWORKS_API_KEY` with pre-run validation | +| `QUICK_REFERENCE.md` | 67 | `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` | `your_fireworks_api_key_here` | +| `verify_deployment.md` | 140 | `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` | `your_fireworks_api_key_here` | +| `START_LOCAL.md` | 217 | `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` | Removed reference, added link to dashboard | +| `DEPLOYMENT_GUIDE.md` | 36 | `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` | Placeholder with instructions | + +### 2. Enhanced .gitignore + +Added comprehensive credential file patterns to prevent future commits: + +```gitignore +# Environment variables and secrets +.env +.env.local +.env.*.local +.env.production +.env.development +.env.test +*.env +*_secrets.json +*_credentials.json +secrets/ +credentials/ +*.key +*.pem +*.p12 +*.crt +*.cer +config/secrets.yml +config/credentials.yml +``` + +### 3. Code Improvements + +**pddl_planner.py:** +- Added `import os` for environment variable access +- Replaced hardcoded API key with `os.getenv("FIREWORKS_API_KEY")` +- Added validation in `call_pddl_model()` to check if API key is set +- Provides helpful error message if key is missing + +**test_pddl_direct.sh:** +- Added environment variable check at script start +- Exits with error message if `FIREWORKS_API_KEY` not set +- Uses `$FIREWORKS_API_KEY` in curl command +- Added instructional comments at top of file + +### 4. Documentation Created + +**SECURITY_MIGRATION.md** - Comprehensive guide covering: +- Phase 1: API key revocation (CRITICAL) +- Phase 2: New key generation +- Phase 3: Git history scrubbing (git-filter-repo & BFG) +- Phase 4: Post-migration verification +- Phase 5: Team communication +- Phase 6: Ongoing security practices +- Complete checklists and audit log template + +--- + +## โš ๏ธ CRITICAL: Required Manual Actions + +### ๐Ÿšจ IMMEDIATE (Must be done ASAP) + +1. **Revoke Exposed API Keys** + ``` + Keys to revoke: + - fw_3ZNkrZnbfKVHhU65bFirkpJr + - fw_3ZHFp8ZR5WeoadXcFcjEKY4z + + Action: Log in to Fireworks AI Dashboard and revoke both keys + Priority: CRITICAL - Do this before any other steps + ``` + +2. **Generate New API Keys** + ``` + Action: Create new API keys in Fireworks AI Dashboard + Apply least privilege principle + Document key purpose and creation date + ``` + +### ๐Ÿ”ง HIGH Priority (Should be done today) + +3. **Git History Scrubbing** + ``` + Tool Options: + - git-filter-repo (recommended) + - BFG Repo-Cleaner (alternative) + + See SECURITY_MIGRATION.md for detailed instructions + Warning: This rewrites git history - requires team coordination + ``` + +4. **Force Push Clean History** + ```bash + git push origin --force --all + git push origin --force --tags + ``` + +5. **Team Communication** + ``` + Notify all team members: + - Repository must be re-cloned + - Old keys are revoked + - New keys must be obtained + - Deadline for compliance + ``` + +### ๐Ÿ“‹ MEDIUM Priority (This week) + +6. **Verify Scrubbing Success** + ```bash + git log --all -S "fw_3ZNkrZnbfKVHhU65bFirkpJr" + git log --all -S "fw_3ZHFp8ZR5WeoadXcFcjEKY4z" + # Should return no results + ``` + +7. **Update Production Environments** + ``` + Railway: Update FIREWORKS_API_KEY variable + Local Dev: Create .env files with new keys + Team: Distribute new keys securely + ``` + +8. **Implement Prevention Measures** + ``` + - Install git-secrets + - Add pre-commit hooks + - Enable GitHub secret scanning + - Schedule quarterly key rotation + ``` + +--- + +## ๐Ÿ“Š Impact Assessment + +### Security Risk Reduction +- **Before:** API keys exposed in codebase and git history (HIGH RISK) +- **After Code Changes:** No keys in current codebase (MEDIUM RISK - history still exposed) +- **After Full Remediation:** Clean codebase and history (LOW RISK) + +### Affected Components +- โœ… Python CLI tool (pddl_planner.py) - now uses environment variables +- โœ… Shell test script (test_pddl_direct.sh) - now validates environment +- โœ… Documentation (6 files) - all placeholders updated +- โœ… Git ignore rules - comprehensive patterns added +- โณ Git history - awaiting scrubbing +- โณ Production deployments - awaiting key rotation + +--- + +## โœ… Verification Checklist + +Run these commands to verify remediation: + +### Code Verification +```bash +# 1. Check no hardcoded keys in code +grep -r "fw_[a-zA-Z0-9]\{20,\}" --include="*.py" --include="*.sh" . +# Should only find commented/placeholder text + +# 2. Verify environment variable usage +grep -n "os.getenv.*FIREWORKS" pddl_planner.py +# Should show line 15: API_KEY = os.getenv("FIREWORKS_API_KEY") + +# 3. Verify shell script validation +grep -n "FIREWORKS_API_KEY" test_pddl_direct.sh +# Should show validation and usage of environment variable + +# 4. Check .gitignore +cat .gitignore | grep -E "(\.env|secret|credential)" +# Should show comprehensive patterns +``` + +### Runtime Verification +```bash +# 1. Test API key validation in Python +python3 pddl_planner.py "test" +# Should error: "FIREWORKS_API_KEY environment variable is not set" + +# 2. Test API key validation in shell +./test_pddl_direct.sh +# Should error: "FIREWORKS_API_KEY environment variable is not set" + +# 3. Test with new key set +export FIREWORKS_API_KEY=new_valid_key_here +python3 pddl_planner.py "test planning problem" +# Should make API call successfully +``` + +--- + +## ๐Ÿ“ž Next Steps + +1. **Review SECURITY_MIGRATION.md** - Read the full guide for detailed instructions +2. **Execute Manual Actions** - Follow the priority order above +3. **Coordinate with Team** - Notify all collaborators of required actions +4. **Document Progress** - Update the audit log in SECURITY_MIGRATION.md +5. **Verify Completion** - Run all verification checks +6. **Implement Prevention** - Set up tools to prevent future exposure + +--- + +## ๐Ÿ“ Files Modified in This Remediation + +### Code Files (2) +- `pddl_planner.py` - Environment variable implementation +- `test_pddl_direct.sh` - Environment variable validation + +### Documentation Files (6) +- `QUICK_REFERENCE.md` - API key placeholder updated +- `verify_deployment.md` - API key placeholder updated +- `START_LOCAL.md` - Removed hardcoded key reference +- `DEPLOYMENT_GUIDE.md` - Added proper instructions +- `SECURITY_MIGRATION.md` - **NEW** - Complete remediation guide +- `SECURITY_REMEDIATION_SUMMARY.md` - **NEW** - This file + +### Configuration Files (1) +- `.gitignore` - Enhanced with credential patterns + +### Total Changes +- **9 files modified/created** +- **2 API keys identified for revocation** +- **0 hardcoded keys remaining in current codebase** +- **Git history scrubbing required** + +--- + +## โšก Quick Reference + +**Exposed Keys (REVOKE IMMEDIATELY):** +``` +fw_3ZNkrZnbfKVHhU65bFirkpJr +fw_3ZHFp8ZR5WeoadXcFcjEKY4z +``` + +**Environment Variable Name:** +``` +FIREWORKS_API_KEY +``` + +**Setting the Variable:** +```bash +# Local development +export FIREWORKS_API_KEY=your_new_key_here + +# Or in .env file (recommended) +echo "FIREWORKS_API_KEY=your_new_key_here" > .env +``` + +**Documentation:** +- Full guide: `SECURITY_MIGRATION.md` +- This summary: `SECURITY_REMEDIATION_SUMMARY.md` + +--- + +**Last Updated:** [Current Date] +**Completed By:** Security Remediation Task +**Next Review:** After git history scrubbing completion diff --git a/START_LOCAL.md b/START_LOCAL.md index a49579a..cc9b639 100644 --- a/START_LOCAL.md +++ b/START_LOCAL.md @@ -214,8 +214,8 @@ npm install ### API Key Invalid 1. Check `FIREWORKS_API_KEY` is set correctly -2. Try the original key: `fw_3ZHFp8ZR5WeoadXcFcjEKY4z` -3. Or get a new key from Fireworks AI +2. Ensure you have a valid API key from Fireworks AI +3. Get a new key from [Fireworks AI Dashboard](https://fireworks.ai/) --- diff --git a/pddl_planner.py b/pddl_planner.py index 3fecccc..447ac6f 100644 --- a/pddl_planner.py +++ b/pddl_planner.py @@ -7,11 +7,12 @@ import argparse import json import sys +import os import requests API_URL = "https://api.fireworks.ai/inference/v1/chat/completions" -API_KEY = "fw_3ZNkrZnbfKVHhU65bFirkpJr" +API_KEY = os.getenv("FIREWORKS_API_KEY") MODEL = "accounts/colin-fbf68a/deployedModels/pddlplanner-turbo-10141406-w5lghxbj" SYSTEM_PROMPT = "You are an expert planning assistant. When given a problem, output a structured plan in PDDL format with actions and explanations." @@ -28,6 +29,11 @@ def call_pddl_model(user_prompt, temperature=0.6, max_tokens=10000): Returns: dict: The API response """ + if not API_KEY: + print("Error: FIREWORKS_API_KEY environment variable is not set.", file=sys.stderr) + print("Please set it using: export FIREWORKS_API_KEY=your_api_key_here", file=sys.stderr) + sys.exit(1) + headers = { "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json" diff --git a/test_pddl_direct.sh b/test_pddl_direct.sh index 3dca98a..230c207 100755 --- a/test_pddl_direct.sh +++ b/test_pddl_direct.sh @@ -1,12 +1,21 @@ #!/bin/bash # Test PDDL API directly with Fireworks AI # Usage: ./test_pddl_direct.sh "Your planning problem here" +# +# IMPORTANT: Set your FIREWORKS_API_KEY environment variable before running: +# export FIREWORKS_API_KEY=your_api_key_here + +if [ -z "$FIREWORKS_API_KEY" ]; then + echo "Error: FIREWORKS_API_KEY environment variable is not set." + echo "Please set it using: export FIREWORKS_API_KEY=your_api_key_here" + exit 1 +fi PROMPT="${1:-I need to move two packages from location A to location B using a truck. The truck starts at location A. Package 1 and Package 2 are both at location A.}" curl -X POST https://api.fireworks.ai/inference/v1/chat/completions \ -H "Content-Type: application/json" \ - -H "Authorization: Bearer fw_3ZHFp8ZR5WeoadXcFcjEKY4z" \ + -H "Authorization: Bearer $FIREWORKS_API_KEY" \ -d @- <