Make web-capture microservice work as expected and integrate it with Telegram bot

[konard]

🎯 Objective

Make web-capture microservice work as expected and integrate it with the Telegram bot.

Fixes: #10

---

📊 Current Status

✅ Completed

• Analyzed web-capture microservice architecture and current state
• Analyzed telegram-bot architecture and integration patterns
• Identified missing integration tests (GitHub, Wikipedia, StackOverflow)
• Created comprehensive implementation plan (IMPLEMENTATION_PLAN.md)
• Created detailed integration design (WEB_CAPTURE_INTEGRATION.md)
• Posted clarifying questions to issue Make web-capture microservice work as expected and integrate it with Telegram bot #10 (comment)

🚧 In Progress

• Awaiting clarification on deployment infrastructure and UX preferences
• Preparing implementation for web-capture repository
• Preparing implementation for telegram-bot repository

---

📋 Analysis Summary

Web-Capture Microservice

Current State:

• ✅ Core API endpoints functional (HTML, Markdown, PNG)
• ✅ Dual browser engine support (Puppeteer + Playwright)
• ✅ Basic tests passing
• ✅ Docker support
• ✅ Integration test for Habr.com (PR Integrate open-source VS Code and open-source GitHub Copilot in browser as developer mode in web app. #9)

What's Missing:

• ❌ Integration tests for GitHub README (issue Standalone iOS/iPadOS application at AppStore #5)
• ❌ Integration tests for Wikipedia (issue Support Cursor by our API Gateway for OpenAI models #8)
• ❌ Integration tests for StackOverflow (issue Automate sync of master plan using GitHub Actions #11)
• ❌ Health check endpoint for monitoring
• ❌ Production deployment configuration
• ❌ Comprehensive error handling

Telegram Bot

Current State:

• ✅ Modular router architecture
• ✅ External service integration pattern
• ✅ Configuration management
• ✅ File handling capabilities

What's Missing:

• ❌ Web-capture service integration
• ❌ URL detection/handling
• ❌ Capture commands

---

🎨 Proposed Implementation

Phase 1: Web-Capture Improvements

1. Add Integration Tests (Issues Standalone iOS/iPadOS application at AppStore #5, Support Cursor by our API Gateway for OpenAI models #8, Automate sync of master plan using GitHub Actions #11)

   • GitHub README test
   • Wikipedia page test
   • StackOverflow question test
   • Verify both Puppeteer and Playwright engines

2. Add Health Check Endpoint

   GET /health
   Response: { status: 'ok', version: '1.0.0', engines: [...] }

3. Improve Error Handling

   • User-friendly error messages
   • Proper HTTP status codes
   • Detailed logging

4. Production Readiness

   • Docker Compose configuration
   • Environment variable documentation
   • Deployment guide

Phase 2: Telegram Bot Integration

1. Configuration

   • Add WEB_CAPTURE_URL to config
   • Add WEB_CAPTURE_ENABLED feature flag
   • Add timeout and engine preferences

2. Service Client

   • HTTP client for web-capture API
   • Methods: get_html(), get_markdown(), get_screenshot()
   • Error handling and retries

3. Command Handlers

   • /capture <url> - Interactive with format selection
   • /markdown <url> - Direct markdown capture
   • /screenshot <url> - Direct screenshot capture

4. User Experience

   • Inline keyboards for format selection
   • Progress messages during capture
   • Smart file handling (message vs document based on size)
   • Friendly error messages

---

📐 Architecture

┌─────────────┐      ┌──────────────┐      ┌─────────────┐
│   Telegram  │─────>│ Telegram Bot │─────>│ Web-Capture │
│    Users    │<─────│   (Python)   │<─────│   Service   │
└─────────────┘      └──────────────┘      └─────────────┘
                            │
                            ├─> API Gateway (existing)
                            ├─> DeepInfra (existing)
                            └─> Web-Capture (new)

User Flow

1. User sends: /capture https://example.com
2. Bot shows inline keyboard: [Markdown] [Screenshot] [Both]
3. User selects format
4. Bot requests from web-capture service
5. Bot sends result (text/file/photo)

---

❓ Clarification Questions (Posted to Issue #10)

Awaiting answers on:

1. Deployment: Should web-capture be deployed to specific infrastructure, or just ensure it's deployment-ready?
2. Integration UX: How should users trigger capture?
   • Auto-detect URLs?
   • Explicit commands only?
   • Both?
3. Output format: How to deliver captured content?
   • Markdown: As text or file?
   • Screenshots: As photo or document?
   • HTML: As file?

---

📄 Documentation

This PR includes two comprehensive documents:

1. IMPLEMENTATION_PLAN.md

• Current state analysis
• 4-phase implementation strategy
• Testing strategy
• Deployment considerations
• Risk mitigation
• Timeline estimate

2. WEB_CAPTURE_INTEGRATION.md

• Detailed architecture design
• Complete code examples for both repositories
• Deployment configurations
• Testing strategy
• User documentation
• Future enhancements roadmap

---

🧪 Testing Strategy

Web-Capture

• Unit tests (existing)
• Integration tests (new - GitHub, Wikipedia, StackOverflow)
• E2E tests with Docker

Telegram Bot

• Unit tests for service client
• Integration tests with mocked web-capture
• E2E tests (optional)

---

✅ Success Criteria

• All web-capture integration tests pass
• Web-capture has health check endpoint
• Web-capture is deployable via Docker Compose
• Telegram bot can capture URLs as markdown
• Telegram bot can capture URLs as screenshots
• Error handling is robust
• Documentation is complete
• Tests verify the integration

---

📦 Next Steps

Once clarification is received:

1. Implement web-capture improvements
2. Create PRs to web-capture repository
3. Implement telegram-bot integration
4. Create PRs to telegram-bot repository
5. Test end-to-end integration
6. Update architecture documentation
7. Mark this PR as ready for review

---

🔗 Related Issues & PRs

Web-Capture Repository:

• Issue Standalone iOS/iPadOS application at AppStore #5: Add test for GitHub README download
• Issue Support Cursor by our API Gateway for OpenAI models #8: Add test for Wikipedia page download
• Issue Automate sync of master plan using GitHub Actions #11: Add test for StackOverflow page download
• PR Integrate open-source VS Code and open-source GitHub Copilot in browser as developer mode in web app. #9: Integration test for Habr article (merged)

This Repository:

• Issue Make web-capture microservice work as expected and integrate it with Telegram bot #10: Make web-capture work and integrate with Telegram bot

---

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

Fixes #10

[konard]

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

📎 Log file uploaded as GitHub Gist (232KB)
🔗 View complete solution draft log

---

Now working session is ended, feel free to review and add any feedback on the solution draft.