plan.md

Project Plan

MCP Screenshot Server Implementation

• Project Setup

  • Initialize Node.js project with TypeScript
  • Set up Biome for linting and formatting
  • Configure basic project structure
  • Set up package.json with necessary dependencies

• Basic Testing Infrastructure

  • Set up test framework (Node.js built-in test runner)
  • Create basic test structure and utilities
  • Add simple unit tests for core functions
  • Create test fixtures (simple HTML pages)
  • Set up test scripts and CI-friendly commands
  • Add basic integration test placeholder

• Docker Integration

  • Create Dockerfile with Playwright browser dependencies
  • Set up docker-compose.yml for development
  • Configure Docker networking for localhost access
  • Add Docker scripts to package.json
  • Create E2E tests for Docker functionality
  • Test container build and deployment

• Core Playwright Service

  • Implement basic screenshot capture service
  • Add support for different viewport sizes
  • Configure WebP output format
  • Add page load waiting strategies
  • Implement error handling and retries

• MCP Server Implementation

  • Set up MCP protocol server with stdio transport
  • Define screenshot tool schema and parameters
  • Implement tool handler for screenshot requests
  • Add base64 image encoding for responses
  • Add parameter validation and defaults

• Configuration & Options

  • Add environment variable support for all default settings
  • Implement configurable quality settings via BROWSERLOOP_DEFAULT_QUALITY
  • Add timeout and retry configuration (BROWSERLOOP_RETRY_COUNT, BROWSERLOOP_RETRY_DELAY)
  • Support custom viewport dimensions (BROWSERLOOP_DEFAULT_WIDTH, BROWSERLOOP_DEFAULT_HEIGHT)
  • Add user agent configuration (BROWSERLOOP_USER_AGENT)
  • Create comprehensive configuration validation with Zod
  • Add retry logic with configurable count and delay
  • Implement graceful fallbacks for invalid environment variables
  • Add configuration tests and documentation

• Advanced Features

  • Full page screenshot support
  • Element-specific screenshot capture
  • Multiple format support (WebP, PNG, JPEG)

• Documentation & Examples

  • Complete API documentation
  • Add usage examples and tutorials
  • Document MCP configuration for AI development tools
  • Create troubleshooting guide
  • Add performance optimization guide

• Error Handling & Reliability

  • Add timeout handling for page loads
  • Implement retry logic for failed screenshots
  • Add browser crash recovery
  • Handle network and Docker errors
  • Add comprehensive logging

• Documentation & Deployment

  • Create README with setup instructions
  • Document MCP configuration for Cursor
  • Add troubleshooting guide
  • Create example usage scenarios
  • Set up CI/CD pipeline (optional)

• Optimization & Polish

  • Optimize Docker image size
  • Add browser session reuse
  • Implement caching strategies
  • Performance testing and tuning
  • Final code review and cleanup

Cookie-Based Authentication Implementation

• Cookie Parameter Support

  • Extend ScreenshotOptions interface with cookies parameter
  • Add cookie validation using Zod schema (array of objects or JSON string)
  • Update MCP tool schema to include cookies parameter
  • Add cookie parsing utilities (JSON string to cookie object array)
  • Implement cookie sanitization (never log cookie values)

• Browser Context Cookie Injection

  • Modify ScreenshotService to accept cookies in options
  • Implement cookie injection before page navigation
  • Add proper cookie domain and path handling
  • Handle cookie format validation and error messages
  • Add timeout handling for cookie-related operations

• Security & Privacy

  • Ensure cookie values are never logged or stored
  • Clear sensitive data from memory after use
  • Add security warnings to documentation
  • Implement proper error messages without exposing cookie data
  • Add input validation to prevent cookie injection attacks

• Testing & Validation

  • Create test fixtures with authentication requirements
  • Add unit tests for cookie parsing and validation
  • Create integration tests with mock authentication
  • Test cookie injection with different domain configurations
  • Add E2E tests with real authenticated scenarios
  • Verify security measures (no cookie leakage in logs)

• Documentation & Examples

  • Document cookie extraction methods (dev tools, browser extensions)
  • Add example cookie formats and usage scenarios
  • Create step-by-step guides for popular sites (GitHub, Gmail)
  • Document browser extension recommendations
  • Add troubleshooting guide for authentication failures
  • Update API documentation with cookie parameter details

• Error Handling & User Experience

  • Add helpful error messages for invalid cookies
  • Implement authentication failure detection
  • Add cookie expiration handling and user guidance
  • Create user-friendly validation error messages
  • Add parameter validation with clear feedback

Cookie Domain Validation & Modern Authentication Support

• Default Cookie Environment Variable Support

  • Implement BROWSERLOOP_DEFAULT_COOKIES environment variable
  • Support both JSON file paths and JSON string values
  • Add automatic file vs string detection (path contains .json or starts with /)
  • Create comprehensive error handling for file operations
  • Add detailed debug logging for cookie loading process

• Cookie Merging System

  • Implement cookie merging between default and request cookies
  • Request cookies override default cookies with same name
  • Add detailed debug logging for merge process
  • Sanitize sensitive values in all log outputs

• Modern Cookie Name Support

  • Fix overly restrictive cookie name validation regex
  • Support RFC 6265 compliant cookie names including dots
  • Add support for __Host- and __Secure- prefixed cookies
  • Test with real-world authentication patterns (Next.js, analytics)
  • Add comprehensive test suite for modern cookie names

• Parent Domain Cookie Support (RFC 6265)

  • Fix domain validation logic for parent domain cookies
  • Implement correct RFC 6265 domain matching rules
  • Support cookies with domain .example.com on subdomain.example.com
  • Add comprehensive domain validation tests
  • Document the fix and provide examples

• __Host- Cookie URL Format Fix

  • Fix __Host- cookie "Invalid cookie fields" error
  • Implement base URL extraction for __Host- cookies (protocol + hostname only)
  • Remove path components from URLs for __Host- cookies
  • Test with real-world Next.js authentication patterns
  • Ensure compatibility with mixed cookie types

• Cookie Expiration & Security Attributes Fix

  • Identify root cause of session cookie issue (missing expires, secure, httpOnly attributes)
  • Create cookie enhancement system to add proper security attributes
  • Implement automatic expiration date assignment (30 days for auth cookies)
  • Fix persistent vs session cookie authentication problems
  • Create cookie enhancement utility script for future updates
  • Verify authentication works with enhanced cookies containing proper attributes
  • FINAL RESOLUTION: All technical cookie issues resolved. Authentication failures are due to server-side session expiration (normal security behavior). Automatic cookie enhancement is working correctly but not needed since cookies are being properly injected. Users need to refresh expired authentication sessions.
  • REMOVED: Manual enhance-cookies.js script removed since automatic enhancement is built into ScreenshotService

• Enhanced Debugging & Documentation

  • Create comprehensive debugging guide for cookie issues
  • Document all common cookie problems and solutions
  • Add step-by-step troubleshooting instructions
  • Update API documentation with new cookie features
  • Create examples for MCP configuration with default cookies

• Browser Extension Cookie Format Support

  • Update cookie validation schema to accept session cookies (expires: -1)
  • Add support for float timestamps from browser extensions (e.g., 1750704030.825311)
  • Respect existing cookie attributes instead of overriding them
  • Ensure enhancement logic only adds missing attributes
  • Test compatibility with real browser extension exports
  • Document browser extension cookie support with examples
  • COMPLETE: Users can now directly use cookie files exported from browser extensions without any manual editing required

• Repository Cleanup

  • Clean up temporary test files with specific identifiers
  • Ensure all examples are suitable for public repository
  • COMPLETE: Repository now contains only generic examples suitable for open source distribution

Linting Error Fixes

• Create proper type definitions to replace any types
• Remove static-only classes (CookieUtils)
• Replace delete operators with undefined assignments
• Replace forEach with for...of loops
• Fix MCP server API issues
• Add missing imports and exports
• Update test files with proper typing

Git Pre-commit Hooks with lint-staged and Husky

• Environment and Version Verification

  • Verify Node.js version compatibility with lint-staged 16.1.0 and Husky 9.1.7
  • Check current git status and ensure repository is clean
  • Verify existing npm scripts (lint and format) work correctly
  • Document current state of .git/hooks directory

• Package Installation

  • Install lint-staged version 16.1.0 as dev dependency
  • Install Husky version 9.1.7 as dev dependency
  • Verify package-lock.json is updated with exact versions
  • Confirm no version conflicts with existing dependencies

• Husky Initialization and Setup

  • Run npx husky init to set up Husky configuration
  • Verify .husky directory is created with correct structure
  • Check that .husky/_/husky.sh script is created
  • Ensure package.json prepare script is added for Husky installation
  • Test Husky setup with npm run prepare

• lint-staged Configuration

  • Create lint-staged configuration in package.json
  • Configure lint-staged to run npm run lint on staged files
  • Configure lint-staged to run npm run format on staged files
  • Set up proper file matching patterns for TypeScript/JavaScript files
  • Add ignore patterns for generated files and node_modules
  • Verify lint-staged config follows v16.1.0 syntax and features

• Pre-commit Hook Setup

  • Create .husky/pre-commit hook script
  • Add npx lint-staged command to pre-commit hook
  • Make pre-commit hook executable (chmod +x)
  • Test pre-commit hook execution manually
  • Verify hook script uses proper shebang and error handling

• Security Considerations

  • Review lint-staged configuration for potential command injection
  • Ensure only trusted commands are executed in hooks
  • Verify file pattern matching doesn't expose sensitive files
  • Add safeguards against hook bypassing (document --no-verify usage)
  • Test hook behavior with various file states (renamed, deleted, etc.)

Cookie Domain Filtering for Multi-Site Support

• Core Cookie Filtering Implementation

  • Add cookie domain filtering function using existing RFC 6265 domain matching logic
  • Filter out cookies that don't match target URL domain before injection
  • Continue screenshot process even when some/all cookies are filtered out
  • Add basic debug logging for filtered cookie count (without values)

• Security Verification

  • Test that filtered cookies never leak to wrong domains
  • Verify RFC 6265 compliance with existing domain matching tests
  • Test with multi-domain cookie files to ensure proper filtering

• Documentation Update

  • Document new domain filtering behavior in API docs
  • Add troubleshooting note about authentication with filtered cookies

Automatic Cookie File Reloading (Fix MCP Caching Issue) ✅ COMPLETE

• Core Configuration Refresh

  • Add refreshConfig() method to ConfigManager class
  • Implement secure cookie file re-reading with proper error handling
  • Add atomic configuration replacement to prevent partial updates
  • Maintain cookie sanitization practices during reload operations

• Automatic File Watching System

  • Implement file watcher for cookie files using Node.js fs.watch() (chose fs.watch over chokidar to avoid external dependencies)
  • Add debouncing to prevent rapid reload attempts during file writes (1000ms debounce delay)
  • Watch all configured cookie file paths automatically
  • Handle file deletion, creation, and modification events
  • Add graceful error handling for watch failures (missing files, permissions)

• Configuration Integration

  • Automatically watch cookie file when BROWSERLOOP_DEFAULT_COOKIES points to a file path
  • Integrate file watcher with ConfigManager initialization
  • Ensure proper cleanup of file watchers on server shutdown
  • Add logging for file watch events (without sensitive data)

• Security & Testing

  • Ensure no cookie values leak to logs during automatic reload operations
  • Add unit tests for file watching mechanisms and debouncing (10 comprehensive tests)
  • Test error handling for corrupted or invalid cookie files during automatic reload
  • Verify backward compatibility with existing configurations
  • Test file watcher cleanup and resource management

• MCP Protocol Compatibility Fix

  • CRITICAL ISSUE IDENTIFIED: Console logging was breaking MCP stdio communication
  • ROOT CAUSE: Console output interfered with JSON-based MCP protocol over stdio
  • SOLUTION: Removed all console.log/console.error from production code paths
  • VERIFICATION: File watching now works silently in MCP environment
  • TESTING: Maintained full debugging capabilities in test mode with mock watchers

• ScreenshotService Dynamic Configuration Fix

  • CRITICAL ISSUE IDENTIFIED: ScreenshotService was using stale configuration snapshot from startup
  • ROOT CAUSE: ScreenshotService held a static reference to serviceConfig.authentication.defaultCookies from initialization time
  • SOLUTION: Modified ScreenshotService.createScreenshotConfig() to fetch fresh authentication config from ConfigManager
  • VERIFICATION: ScreenshotService now uses config.getAuthenticationConfig() for each screenshot operation
  • TESTING: Verified that configuration changes are immediately reflected in new screenshot requests

• File Watcher Recreation After Rename Events Fix

  • CRITICAL ISSUE IDENTIFIED: File watcher stopped working after first change due to editor file replacement behavior
  • ROOT CAUSE: Editors create temp files and rename them to replace originals, leaving fs.watch() watching dead inodes
  • SOLUTION: Added watcher recreation logic for 'rename' events with proper cleanup and delayed recreation
  • IMPLEMENTATION: Created recreateWatcher() method with file existence checks and 100ms delay for file operations
  • VERIFICATION: File watcher now survives multiple file edits and continues monitoring indefinitely

• Documentation & Fallback

  • Document automatic file watching capabilities in API documentation
  • Add troubleshooting guide for file watching issues
  • Manual configuration refresh available via config.refreshConfig() method for edge cases
  • Updated documentation for file-based logging: README.md and Cookie Authentication Guide now reflect new /tmp/browserloop.log logging
  • Removed BROWSERLOOP_SILENT environment variable: No longer needed since file logging doesn't interfere with MCP protocol
  • COMPLETE: Automatic file watching resolves MCP caching issues and works correctly in production MCP environments

Console Log Reading Tool Implementation

• MCP Tool Setup

  • Add read_console tool to MCP server schema
  • Define tool parameters (URL, timeout, sanitize option)
  • Create tool handler in MCP server

• Console Log Service

  • Create ConsoleLogService class following existing architecture
  • Implement Playwright console event listener (console.log, warn, error)
  • Collect logs from page creation until load complete + 2 second delay
  • Format logs as JSON with metadata (timestamp, level, message, args)

• Security & Configuration

  • Add BROWSERLOOP_SANITIZE_LOGS config option (default: true, opt-out)
  • Implement sensitive data masking (API keys, tokens, emails, URLs with auth)
  • Mask only sensitive parts, keep message structure intact
  • Add basic log size limits to prevent memory issues

• Integration & Testing

  • Integrate with existing authentication (cookie support)
  • Add unit tests for log collection and sanitization
  • Test with both screenshot and console tools together
  • Update documentation with console log examples
  • Update README with console log tool description and usage

NPX Distribution Implementation

• Check npm package name availability (browserloop or scoped alternative)
• Configure package.json for npm publishing (bin, files, publishConfig)
• Add runtime browser detection with helpful error messaging (stderr + log file) - REMOVED: Relying on documentation instead of complex detection
• Enhance README with clear browser installation instructions for MCP users
• Pre-build TypeScript to include dist/ folder in package
• Update CLI to handle npx usage patterns and help text
• Test complete npx workflow: npx -y browserloop@latest
• Create npx-specific documentation and setup guides
• Security audit: dependencies, browser installation, package integrity
• Publish to npm registry with proper versioning
• Set up automated CI/CD publishing pipeline
  • Add npm publishing to GitHub Actions release workflow
  • Configure NPM_TOKEN secret requirement
  • Add pre-publish validation and post-publish verification steps
  • Create automated versioning scripts (patch/minor/major)
  • Document complete release automation process
• Update README with npx installation instructions
  • Add npm package badges and links
  • Enhance Quick Start with global installation option
  • Update MCP configuration examples for both npx and global install
  • Add package information section with npm registry links
  • Improve troubleshooting with npm-specific diagnostics
  • Add update instructions for both installation methods
  • Include Docker alternative for containerized environments