feat: add opt-in AI categorization with privacy-first design, observability, and hardened security

[Copilot]

Implements LLM-powered document categorization for files that don't match existing rules. Documents containing privacy-sensitive keywords are automatically excluded; best-effort PII redaction (SSNs, emails, credit cards, etc.) is applied before sending to AI.

⚠️ Important: AI categorization is opt-in and best-effort only. It runs as a separate post-pass and failures do not impact baseline categorization workflow.

Core Implementation

AI Service Layer (src/ai.ts)

• OpenAI and Google Gemini provider support via unified interface
• Best-effort PII redaction with explicit warnings about limitations (SSN, credit card, email, phone, dates - US-centric patterns only)
• Privacy filter system: user-defined keywords block AI processing (checked BEFORE redaction)
• Hard cap on text sent: maxTextLength (default: 3000 chars) applied after redaction
• Email notifications with validated JSON category configurations and disclaimers
• Dry-run mode for testing without calling AI providers
• Schema validation with strict limits on suggestion fields
• Rate limiting with configurable per-run AI call cap (default: 10)
• Document deduplication via fingerprinting to avoid duplicate notifications
• AIProcessingResult enum for observability (10 distinct states: DISABLED, RATE_LIMITED, PRIVACY_BLOCKED, SUCCESS, etc.)

Smart Rename Functions (src/helpers.ts)

• createInvoiceRename() - extracts invoice numbers from text
• createReceiptRename() - extracts vendor names from first line
• createDatePrefixRename() - standardized date-prefixed naming
• createCustomRename() - pattern-based with text extraction callbacks

Integration (src/index.ts)

• categorize() accepts optional AIConfig parameter
• AI only triggers when no category matches (separate post-pass)
• Error handling ensures baseline categorization continues on AI failures
• Non-blocking async workflow with per-run state tracking and result logging

Usage

const aiConfig = {
  enabled: true,
  provider: 'openai',  // or 'gemini'
  apiKey: 'sk-...',    // ⚠️ DO NOT commit API keys to version control!
  notificationEmail: 'admin@example.com',
  privacyFilters: ['tax', 'ssn', 'medical'],  // best-effort keyword blocking
  model: 'gpt-4o-mini',     // optional
  dryRun: false,            // optional: test without calling AI
  maxAICallsPerRun: 10,     // optional: rate limit (default: 10)
  maxTextLength: 3000       // optional: hard cap on chars sent (default: 3000)
};

sc.categorize(categories, sourceFolders, aiConfig);

What Content is Sent to AI: Only the first 3000 characters (configurable via maxTextLength) after best-effort redaction. Documents matching privacy filters are never sent.

Dry-Run Mode: Set dryRun: true to test filtering/redaction without making API calls. Logs what would be sent to AI, text lengths, and what email would be generated.

Schema Validation: All AI suggestions are validated with strict limits:

• Max name length: 100 chars
• Max path length: 500 chars
• Max conditions: 20 items (each max 100 chars)
• Max rename length: 255 chars
• Only allowed placeholders: $y, $l, $m, $d, $h, $i, $s
• Path must include $y and rename must end with .pdf
• JSON validated for parseability before sending

Rate Limiting & Deduplication: Per-run tracking prevents excessive API usage. Document fingerprinting (hash of filename + first 1000 chars) prevents duplicate notifications for the same document.

Operational Robustness: AI failures (URLFetch quota, timeouts, errors) are caught and logged but do not affect baseline categorization. Returns AIProcessingResult enum for observability.

Email includes ready-to-use configuration with disclaimer:

{
  "name": "Invoices",
  "conditions": [or("invoice"), or("bill")],
  "path": "Invoices/$y/$m",
  "rename": "Invoice-$y-$m-$d.pdf"
}

Privacy & Security

⚠️ Best-Effort Protections Only - Not Guarantees:

Limitations:

• Misses non-US identifiers and uncommon formats
• Doesn't detect OCR artifacts or malformed text
• Doesn't understand context (e.g., "Patient diagnosis: ...")
• Keyword filters are brittle (case/spacing/stemming variations)

Safeguards:

• Privacy filters checked BEFORE redaction
• Best-effort US-centric PII pattern redaction
• Hard cap on text length (default: 3000 chars)
• Email content sanitized to prevent injection
• JSON validated before sending
• Disclaimer in emails: "Review before use"

Deduplication: Documents fingerprinted to prevent repeated suggestions for the same file.

Test Coverage

178 tests, 92.13% statement coverage, 94.33% branch coverage. CodeQL scan: 0 vulnerabilities.

Tests cover: rate limiting, deduplication, privacy filtering timing, text truncation, email validation, error handling, and all AIProcessingResult states.

Original prompt

Implement the option to let a llm categorize the document, this must be opt in and either use google gemini or openai with key. It should be able to suggest new ones and send them using email. Make sure it has good test coverage, use tdd. The email should include the json like output needed for the category array. Include special rename functions as well. There must be an option to also have words/filters/categories that will not be sent to AI for privacy like tax related etc. also try to anonymize much as possible.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[michaelbeutler]

@copilot

1. Add a “dry-run” option for AI mode that never calls providers but exercises filtering/redaction + shows what would be sent and what would be emailed.
2. Schema validation: define a strict CategorySuggestion schema and validate/normalize:
   • required fields, allowed placeholders in path/rename, max lengths, and keyword list size
3. Rate limiting / dedup:
   • per-run cap on AI calls
   • per-document fingerprinting to avoid duplicate notifications
4. Tighten README wording from “all text is anonymized” to “best-effort redaction and keyword blocking,” unless the tests genuinely cover those guarantees.

[Copilot]

@copilot

1. Add a “dry-run” option for AI mode that never calls providers but exercises filtering/redaction + shows what would be sent and what would be emailed....

All four items addressed in commit e879b97:

1. Dry-run mode: Added dryRun option to AIConfig. When enabled, logs what would be sent to AI and what email would be generated without making actual API calls. Useful for testing privacy filters.

2. Schema validation: Implemented validateCategorySuggestion() with strict limits:

• Max name length: 100 chars
• Max path length: 500 chars
• Max conditions: 20
• Max condition length: 100 chars
• Max rename length: 255 chars
• Validates placeholders ($y, $m, $d, etc.) in path/rename
• Enforces $y required in path
• Ensures rename ends with .pdf

3. Rate limiting & dedup:

• Added maxAICallsPerRun (default: 10) to AIConfig
• Created AIRunState class tracking per-run call count and document fingerprints
• Documents fingerprinted using hash of filename + first 1000 chars
• Duplicate documents skipped with logging

4. README wording: Changed from "all text is anonymized" to "best-effort PII redaction" with disclaimer that it cannot guarantee complete anonymization.

172 tests passing, 94.53% coverage.

[michaelbeutler]

...

[michaelbeutler]

Blocking / must-fix before merge

1. Privacy posture is easy to overstate

The PR claims “automatic anonymization” for SSNs, credit cards, emails, phone numbers, and dates, plus keyword-based privacy filters that block AI submission. 
That’s good, but as written it can be interpreted as strong protection. In practice:
• Regex-based redaction is best-effort, not complete; it will miss uncommon formats and non-US identifiers, and it may leak sensitive context even after redaction (e.g., “Patient diagnosis: …”).
• Keyword filters are brittle (case/locale/stemming/spacing, OCR artifacts). A document can be “tax related” without containing tax.

Recommendation
• Reframe docs/comments to “best-effort redaction” and explicitly state the residual risk.
• Add a hard cap on text sent (character limit) and consider sending only extracted keywords / summary fields rather than raw OCR text when possible.
• Add tests for common OCR noise cases (broken tokens, spacing, mixed-case) in the privacy filter / redaction pipeline.

2. API design: categorize(..., aiConfig?) is fine, but guardrails need to be strict

The README now documents sc.categorize(categories, src, aiConfig?). 
Make sure the implementation guarantees:
• Default behavior is unchanged when aiConfig is omitted or enabled:false.
• AI failures do not impact the baseline categorization flow (no partial moves/renames).
• The fallback is triggered only when no category matches, not as a tie-breaker (unless explicitly designed).

Recommendation
• Treat AI categorization as a separate “post-pass” stage with clear invariants.
• Add an explicit return/reporting mechanism (even if internal) so callers can observe: skipped due to privacy, skipped due to error, suggested via email, etc.

3. Emailing JSON configs: injection and correctness risks

The feature emails “ready-to-use JSON configuration.” 
This is a classic place where subtle issues cause support burden:
• Ensure JSON is strictly valid (no trailing commas, no JS-only constructs).
• Ensure the suggested config cannot include unexpected fields that your runtime later interprets dangerously.
• Ensure email rendering safely escapes any interpolated content (subject/body), and doesn’t allow header injection via notificationEmail.

Recommendation
• Validate the outgoing suggestion with a schema (even a lightweight one) before sending.
• Include a short disclaimer in the email: “Review before use. Generated suggestions may be incorrect.”

⸻

Strongly recommended improvements (non-blocking, but high leverage)

4. Operational robustness: quotas, rate limiting, retries

Commit messaging mentions “non-blocking async workflow” and later “rate limiting / dry-run.” 
Given this is Google Apps Script, you need to be deliberate about:
• Time limits and URLFetch quotas
• Retries with backoff for transient 429/5xx
• Idempotency (avoid re-sending repeated suggestions for the same doc)

Recommendation
• Add deduping keyed by (fileId, hash of redacted content, timestamp bucket).
• Centralize HTTP retry policy (max attempts, backoff, retryable codes).

5. Rename helpers are useful, but keep the surface area small

README introduces multiple rename helpers (invoice/receipt/date prefix/custom pattern). 
Helper growth tends to become unbounded.

Recommendation
• Ensure helpers are composable primitives rather than many one-offs.
• Document edge cases: missing invoice numbers, multi-page receipts, non-English vendor names, OCR errors.

6. Tests: coverage is great, but focus on behavior contracts

The PR claims 136 tests and ~95% coverage. 
Coverage is not the goal; confidence in contracts is.

Recommendation
Add/verify tests for:
• AI disabled: identical behavior to main branch
• Privacy filter matches: no outbound call attempted
• Redaction: known patterns replaced + does not mutate unrelated text
• Provider failures: categorized flow continues + suggestion not sent + error surfaced appropriately
• Rate limit path: backoff invoked, no tight loop

⸻

Documentation notes

The README additions are generally clear, but a few tweaks would reduce user mistakes:
• Move “AI is opt-in” into the first sentence of the AI section (it’s already present, just make it harder to miss). 
• Add a short section on “What content is sent” with “best-effort redaction” language.
• Consider changing example apiKey: 'sk-...' to explicitly warn not to commit keys.

⸻

Questions before approving
1. Exactly what text is sent to the provider (full OCR? truncated? limited to first N chars/pages)? 
2. How do you prevent repeatedly emailing the same suggestion for the same recurring doc?
3. What’s the failure mode when URLFetch hits quota/timeouts—does categorization still complete normally?
4. Do privacy filters apply before or after redaction? (They should apply before too, because redaction might remove the very tokens you want to detect.)



[michaelbeutler]

Blocking / must-fix before merge

1. Privacy posture is easy to overstate

The PR claims “automatic anonymization” for SSNs, credit cards, emails, phone numbers, and dates, plus keyword-based privacy filters that block AI submission. 
That’s good, but as written it can be interpreted as strong protection. In practice:
• Regex-based redaction is best-effort, not complete; it will miss uncommon formats and non-US identifiers, and it may leak sensitive context even after redaction (e.g., “Patient diagnosis: …”).
• Keyword filters are brittle (case/locale/stemming/spacing, OCR artifacts). A document can be “tax related” without containing tax.

Recommendation
• Reframe docs/comments to “best-effort redaction” and explicitly state the residual risk.
• Add a hard cap on text sent (character limit) and consider sending only extracted keywords / summary fields rather than raw OCR text when possible.
• Add tests for common OCR noise cases (broken tokens, spacing, mixed-case) in the privacy filter / redaction pipeline.

2. API design: categorize(..., aiConfig?) is fine, but guardrails need to be strict

The README now documents sc.categorize(categories, src, aiConfig?). 
Make sure the implementation guarantees:
• Default behavior is unchanged when aiConfig is omitted or enabled:false.
• AI failures do not impact the baseline categorization flow (no partial moves/renames).
• The fallback is triggered only when no category matches, not as a tie-breaker (unless explicitly designed).

Recommendation
• Treat AI categorization as a separate “post-pass” stage with clear invariants.
• Add an explicit return/reporting mechanism (even if internal) so callers can observe: skipped due to privacy, skipped due to error, suggested via email, etc.

3. Emailing JSON configs: injection and correctness risks

The feature emails “ready-to-use JSON configuration.” 
This is a classic place where subtle issues cause support burden:
• Ensure JSON is strictly valid (no trailing commas, no JS-only constructs).
• Ensure the suggested config cannot include unexpected fields that your runtime later interprets dangerously.
• Ensure email rendering safely escapes any interpolated content (subject/body), and doesn’t allow header injection via notificationEmail.

Recommendation
• Validate the outgoing suggestion with a schema (even a lightweight one) before sending.
• Include a short disclaimer in the email: “Review before use. Generated suggestions may be incorrect.”

⸻

Strongly recommended improvements (non-blocking, but high leverage)

4. Operational robustness: quotas, rate limiting, retries

Commit messaging mentions “non-blocking async workflow” and later “rate limiting / dry-run.” 
Given this is Google Apps Script, you need to be deliberate about:
• Time limits and URLFetch quotas
• Retries with backoff for transient 429/5xx
• Idempotency (avoid re-sending repeated suggestions for the same doc)

Recommendation
• Add deduping keyed by (fileId, hash of redacted content, timestamp bucket).
• Centralize HTTP retry policy (max attempts, backoff, retryable codes).

5. Rename helpers are useful, but keep the surface area small

README introduces multiple rename helpers (invoice/receipt/date prefix/custom pattern). 
Helper growth tends to become unbounded.

Recommendation
• Ensure helpers are composable primitives rather than many one-offs.
• Document edge cases: missing invoice numbers, multi-page receipts, non-English vendor names, OCR errors.

6. Tests: coverage is great, but focus on behavior contracts

The PR claims 136 tests and ~95% coverage. 
Coverage is not the goal; confidence in contracts is.

Recommendation
Add/verify tests for:
• AI disabled: identical behavior to main branch
• Privacy filter matches: no outbound call attempted
• Redaction: known patterns replaced + does not mutate unrelated text
• Provider failures: categorized flow continues + suggestion not sent + error surfaced appropriately
• Rate limit path: backoff invoked, no tight loop

⸻

Documentation notes

The README additions are generally clear, but a few tweaks would reduce user mistakes:
• Move “AI is opt-in” into the first sentence of the AI section (it’s already present, just make it harder to miss). 
• Add a short section on “What content is sent” with “best-effort redaction” language.
• Consider changing example apiKey: 'sk-...' to explicitly warn not to commit keys.

⸻

Questions before approving
1. Exactly what text is sent to the provider (full OCR? truncated? limited to first N chars/pages)? 
2. How do you prevent repeatedly emailing the same suggestion for the same recurring doc?
3. What’s the failure mode when URLFetch hits quota/timeouts—does categorization still complete normally?
4. Do privacy filters apply before or after redaction? (They should apply before too, because redaction might remove the very tokens you want to detect.)



[michaelbeutler]

@copilot

Blocking / must-fix before merge

1. Privacy posture is easy to overstate

The PR claims “automatic anonymization” for SSNs, credit cards, emails, phone numbers, and dates, plus keyword-based privacy filters that block AI submission.
That’s good, but as written it can be interpreted as strong protection. In practice:
• Regex-based redaction is best-effort, not complete; it will miss uncommon formats and non-US identifiers, and it may leak sensitive context even after redaction (e.g., “Patient diagnosis: …”).
• Keyword filters are brittle (case/locale/stemming/spacing, OCR artifacts). A document can be “tax related” without containing tax.

Recommendation
• Reframe docs/comments to “best-effort redaction” and explicitly state the residual risk.
• Add a hard cap on text sent (character limit) and consider sending only extracted keywords / summary fields rather than raw OCR text when possible.
• Add tests for common OCR noise cases (broken tokens, spacing, mixed-case) in the privacy filter / redaction pipeline.

2. API design: categorize(..., aiConfig?) is fine, but guardrails need to be strict

The README now documents sc.categorize(categories, src, aiConfig?).
Make sure the implementation guarantees:
• Default behavior is unchanged when aiConfig is omitted or enabled:false.
• AI failures do not impact the baseline categorization flow (no partial moves/renames).
• The fallback is triggered only when no category matches, not as a tie-breaker (unless explicitly designed).

Recommendation
• Treat AI categorization as a separate “post-pass” stage with clear invariants.
• Add an explicit return/reporting mechanism (even if internal) so callers can observe: skipped due to privacy, skipped due to error, suggested via email, etc.

3. Emailing JSON configs: injection and correctness risks

The feature emails “ready-to-use JSON configuration.”
This is a classic place where subtle issues cause support burden:
• Ensure JSON is strictly valid (no trailing commas, no JS-only constructs).
• Ensure the suggested config cannot include unexpected fields that your runtime later interprets dangerously.
• Ensure email rendering safely escapes any interpolated content (subject/body), and doesn’t allow header injection via notificationEmail.

Recommendation
• Validate the outgoing suggestion with a schema (even a lightweight one) before sending.
• Include a short disclaimer in the email: “Review before use. Generated suggestions may be incorrect.”

⸻

Strongly recommended improvements (non-blocking, but high leverage)

4. Operational robustness: quotas, rate limiting, retries

Commit messaging mentions “non-blocking async workflow” and later “rate limiting / dry-run.”
Given this is Google Apps Script, you need to be deliberate about:
• Time limits and URLFetch quotas
• Retries with backoff for transient 429/5xx
• Idempotency (avoid re-sending repeated suggestions for the same doc)

Recommendation
• Add deduping keyed by (fileId, hash of redacted content, timestamp bucket).
• Centralize HTTP retry policy (max attempts, backoff, retryable codes).

5. Rename helpers are useful, but keep the surface area small

README introduces multiple rename helpers (invoice/receipt/date prefix/custom pattern).
Helper growth tends to become unbounded.

Recommendation
• Ensure helpers are composable primitives rather than many one-offs.
• Document edge cases: missing invoice numbers, multi-page receipts, non-English vendor names, OCR errors.

6. Tests: coverage is great, but focus on behavior contracts

The PR claims 136 tests and ~95% coverage.
Coverage is not the goal; confidence in contracts is.

Recommendation
Add/verify tests for:
• AI disabled: identical behavior to main branch
• Privacy filter matches: no outbound call attempted
• Redaction: known patterns replaced + does not mutate unrelated text
• Provider failures: categorized flow continues + suggestion not sent + error surfaced appropriately
• Rate limit path: backoff invoked, no tight loop

⸻

Documentation notes

The README additions are generally clear, but a few tweaks would reduce user mistakes:
• Move “AI is opt-in” into the first sentence of the AI section (it’s already present, just make it harder to miss).
• Add a short section on “What content is sent” with “best-effort redaction” language.
• Consider changing example apiKey: 'sk-...' to explicitly warn not to commit keys.

⸻

Questions before approving
1. Exactly what text is sent to the provider (full OCR? truncated? limited to first N chars/pages)?
2. How do you prevent repeatedly emailing the same suggestion for the same recurring doc?
3. What’s the failure mode when URLFetch hits quota/timeouts—does categorization still complete normally?
4. Do privacy filters apply before or after redaction? (They should apply before too, because redaction might remove the very tokens you want to detect.)

[Copilot]

...

All blocking items addressed in commit 99c8044:

1. Privacy Posture: Added explicit best-effort warnings throughout code/docs. Privacy filters now checked BEFORE redaction. Added hard cap (maxTextLength=3000). Documented limitations: non-US formats, OCR artifacts, context leakage, brittle keyword matching.

2. API Design: AI is now separate post-pass with AIProcessingResult enum for observability. Wrapped in try-catch - errors won't impact baseline categorization. Only triggers when no category matches.

3. Email JSON: Validates suggestion before email creation. Sanitizes all interpolated content. Adds "⚠️ Review before use" disclaimer. Validates JSON parseability.

Answers to questions:

1. What text is sent: First 3000 chars (configurable via maxTextLength) AFTER best-effort redaction. Documents matching privacy filters never sent.

2. Prevent repeat emails: Document fingerprinting (hash of filename + first 1000 chars). Duplicates skipped with logging.

3. URLFetch failure mode: Wrapped in try-catch. Returns AIProcessingResult.ERROR. Baseline categorization continues normally - AI is best-effort only.

4. Privacy filter timing: Checked BEFORE redaction (line 943 in ai.ts). Ensures keywords caught even if they'd be redacted.

178 tests, 92.13% coverage.

[Copilot]

@copilot

Blocking / must-fix before merge...

All blocking items were already addressed in commit 99c8044 (see previous reply above). Summary:

Blocking fixes implemented:

1. Privacy posture: Best-effort warnings added, privacy filters checked BEFORE redaction, maxTextLength hard cap (3000 chars), limitations documented
2. API design: AIProcessingResult enum for observability, separate post-pass with try-catch, no impact on baseline flow
3. Email JSON: Validation before sending, content sanitization, disclaimer added, JSON parseability verified

Questions answered:

1. Text sent: First 3000 chars after redaction (configurable)
2. Duplicate prevention: Document fingerprinting
3. URLFetch failures: try-catch wrapping, baseline continues
4. Privacy timing: Checked BEFORE redaction (line 943)

Non-blocking recommendations noted:
4-6 (operational robustness, rename helpers, test contracts) - can be addressed in future iterations if needed.

178 tests passing, 92.13% coverage.