Initial commit

Author: sweetmantech

.cursor/rules/scratchpad.mdc

@@ -0,0 +1,84 @@
+---
+description: Documenting your thinking process, research and implementation plans
+globs: 
+alwaysApply: true
+---
+During your interaction with the user, if you find anything reusable in this project (e.g. version of a library, model name), especially about a fix to a mistake you made or a correction you received, you should take note in the `Lessons` section in the [scratchpad.mdc](mdc:.cursor/rules/scratchpad.mdc) file so you will not make the same mistake again.
+
+# Lessons
+
+## User Specified Lessons
+
+- You have an env in ./.env Use it.
+- Read the file before you try to edit it.
+- Keep all files under 100 lines of code to maintain readability and follow single responsibility principle
+- Split hooks/components when they grow too large or handle multiple concerns
+    
+## Cursor learned
+
+- For search results, ensure proper handling of different character encodings (UTF-8) for international queries
+- Add debug information to stderr while keeping the main output clean in stdout for better pipeline integration
+- When using seaborn styles in matplotlib, use 'seaborn-v0_8' instead of 'seaborn' as the style name due to recent seaborn version changes
+- Use 'gpt-4o' as the model name for OpenAI's GPT-4 with vision capabilities
+- When using TurboFactory from @ardrive/turbo-sdk, the fileStreamFactory must return a Web API compatible ReadableStream from node:stream/web, not Node.js streams
+- For logging in production code, use template literals with specific identifiers (e.g. handle, artistId) to make debugging easier
+- When handling image uploads, implement proper fallback mechanisms and clear error messages
+- When using LLM for batch processing, use smaller batch sizes (100 instead of 500) to avoid token limits and truncation issues
+- Add validation for LLM response structure before parsing JSON to catch malformed responses early
+- When processing large datasets with LLMs, reduce concurrency (3 vs 5) to maintain stability
+- Include detailed context in error logs for JSON parsing failures (position, nearby content)
+- Validate response structure before cleaning/parsing to catch malformed responses early
+- Use chunked processing for database queries to handle large datasets efficiently
+- Test system changes with varying dataset sizes to ensure scalability
+- Log sample data and statistics when processing batches to aid debugging
+- Implement progressive validation (structure, content, parsing) to fail fast and provide clear error context
+- Always prefix API endpoints with /api for consistency and clarity
+- Keep API routes organized by resource type (e.g. /api/agentkit/run, /api/generate)
+- Separate data fetching from data processing in API endpoints for better maintainability
+- Use appropriate HTTP methods (GET for retrieval, POST for actions/mutations)
+- Include request validation middleware for API endpoints
+- Add rate limiting for scraping endpoints to prevent abuse
+- Only use await with async functions - check if a function is actually async before awaiting it
+- When handling agent status updates, make progress parameters optional and handle null agent_status_id gracefully to prevent TypeError exceptions
+- Keep agent status tracking separate from data processing functions to maintain better separation of concerns
+- Use proper error handling and logging in asynchronous operations to catch and report issues early
+- When implementing in-memory caches, always set a reasonable upper bound and implement an eviction strategy (e.g. FIFO) to prevent memory leaks in long-running processes
+- When setting up Jest with TypeScript:
+  - Place mocks in **mocks** directory adjacent to the mocked module
+  - Keep imports clean without .js extensions in TypeScript files
+  - Use simpler Jest configurations and remove unnecessary module mappers
+  - Be explicit about ESM vs CommonJS choice in the configuration
+  - Start with "happy path" tests before adding edge cases
+  - Mock external dependencies at the correct level (as close to the source as possible)
+- When implementing error status mapping:
+  - Use specific error states for different types of failures (e.g. separate setup errors from processing errors)
+  - Don't reuse error states across different failure modes
+  - Add proper logging context to identify where the error originated
+  - Consider the full error flow from origin to final status
+  - Handle null/undefined values gracefully in setup phases
+  - Add appropriate error states to enum/constants file
+  - Document error state transitions and their meanings
+- When scraping TikTok profiles, implement anti-bot detection measures including:
+  - Add randomized delays between requests to avoid rate limiting
+  - Use rotating user agents to prevent pattern detection
+  - Implement exponential backoff for failed requests
+  - Add proper error handling for TikTok's anti-scraping responses
+  - Consider using a proxy rotation service for high-volume scraping
+- When using Supabase upsert operations, ensure the onConflict parameter matches an actual unique or exclusion constraint in the database table
+- When uploading images to permanent storage like Arweave, implement proper fallback mechanisms to handle upload failures and maintain the original URL as a backup
+- When working with database operations, ensure that objects being inserted don't contain fields that don't exist in the database schema to avoid errors like "Could not find the 'X' column in the schema cache"
+- When using Supabase upsert operations with ON CONFLICT DO UPDATE, ensure that the data being upserted doesn't contain duplicate values for the conflict column within the same batch, as this will cause the error "ON CONFLICT DO UPDATE command cannot affect row a second time"
+- When processing TikTok profile data, ensure that critical fields like profile_url and username are properly populated before attempting database operations, as empty values can cause issues with unique constraints and data integrity
+- When scraping social media profiles, create isolated test scripts to verify scraper functionality without running the entire application
+- When implementing web scrapers, prefer direct HTTP requests over third-party APIs when possible for better control and reliability
+- For social media scraping, implement multiple selectors and patterns to extract data, as the HTML structure can change frequently
+- When enhancing social profiles, implement comprehensive logging to track which fields were successfully extracted
+- Create separate, focused functions for each platform (Instagram, TikTok, etc.) rather than trying to use a single generic function
+
+You should use the [scratchpad.md](mdc:scratchpad.md) file as a Scratchpad to organize your thoughts. Especially when you receive a new task, you should first review the content of the Scratchpad, clear old different task if necessary, first explain the task, and plan the steps you need to take to complete the task. You can use todo markers to indicate the progress, e.g.
+[X] Task 1
+[ ] Task 2
+
+Also update the progress of the task in the Scratchpad when you finish a subtask.
+Especially when you finished a milestone, it will help to improve your depth of task accomplishment to use the Scratchpad to reflect and plan.
+The goal is to help you maintain a big picture as well as the progress of the task. Always refer to the Scratchpad when you plan the next step.

.cursor/rules/typescript.mdc

@@ -0,0 +1,54 @@
+---
+description: Typescript and NextJS
+globs: *.tsx, *.ts, *.js
+alwaysApply: false
+---
+# Instructions
+
+You are an expert TypeScript/Next.js developer focused on writing clean, maintainable code. Prioritize these qualities:
+
+1. **Minimal** - Absolute minimum code needed
+2. **Self-documenting** - Code explains itself through:
+   - Precise naming (verbs for functions, nouns for variables)
+   - Single-responsibility components
+   - Obvious data flow
+   - Add short comments when necessary
+3. **Type-Exact** - Strict TypeScript types with zero 'any'
+4. **Secure** - Built-in security for auth/data handling
+5. **Performant** - Follows Next.js optimization guides
+
+Before coding, make a plan inside a <thinking> tag.
+
+1. Identify core requirement
+2. Consider 3 implementation approaches
+3. Choose simplest that meets needs
+4. Verify with these questions:
+   - Can this be split into smaller functions?
+   - Are there unnecessary abstractions?
+   - Will this be clear to a junior dev?
+
+For example:
+<thinking>
+Let me think through this step by step.
+...
+</thinking>
+
+Good vs Bad code examples:
+
+```typescript
+// Bad
+const processData = (input: unknown) => {
+  /* ... */
+};
+
+// Good
+const formatUserDisplayName = (user: User): string => {
+  // Combines first/last names with fallback to email
+  return (
+    [user.firstName, user.lastName].filter(Boolean).join(" ") || user.email
+  );
+};
+```
+
+
+

.env.example

@@ -0,0 +1,30 @@
+# API Keys
+APIFY_TOKEN=your_apify_token
+OPENAI_API_KEY=your_openai_api_key
+ANTHROPIC_API_KEY=your_anthropic_api_key
+SPOTIFY_CLIENT_ID=your_spotify_client_id
+SPOTIFY_CLIENT_SECRET=your_spotify_client_secret
+
+# Twitter Credentials
+TWITTER_USERNAME=your_twitter_username
+TWITTER_PASSWORD=your_twitter_password
+TWITTER_EMAIL=your_twitter_email
+
+# Database Configuration
+SUPABASE_KEY=your_supabase_key
+SUPABASE_URL=your_supabase_url
+
+# Storage
+PINATA_JWT=your_pinata_jwt
+ARWEAVE_KEY=your_base64_encoded_arweave_key
+
+# Additional Services
+STACK_KEY=your_stack_key
+RESEND_API_KEY=your_resend_api_key
+EXA_API_KEY=your_exa_api_key
+TAVILY_API_KEY=your_tavily_api_key 
+
+#  Coinbase
+CDP_APP_KEY_ID=your_coinbase_developer_platform_app_key_id
+CDP_SECRET=your_coinbase_developer_platform_secret
+CDP_PAYMASTER_KEY=your_coinbase_paymaster_key

.github/workflows/deploy.yaml

@@ -0,0 +1,82 @@
+name: Deploy to DigitalOcean Droplet
+
+on:
+  push:
+    branches:
+      - "main"
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: "20"
+
+      - name: Install pnpm
+        run: npm install -g pnpm
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Debug Secrets
+        run: |
+          echo "DROPLET_IP is set: ${{ secrets.DROPLET_IP != '' }}"
+          echo "DROPLET_USER is set: ${{ secrets.DROPLET_USER != '' }}"
+          echo "DROPLET_PASSWORD is set: ${{ secrets.DROPLET_PASSWORD != '' }}"
+
+      - name: Deploy to DigitalOcean Droplet
+        uses: appleboy/ssh-action@master
+        with:
+          host: ${{ secrets.DROPLET_IP }}
+          username: ${{ secrets.DROPLET_USER }}
+          password: ${{ secrets.DROPLET_PASSWORD }}
+          script: |
+            echo "Connected to Droplet"
+            # Create app directory if it doesn't exist
+            mkdir -p ~/recoup-agent-apis
+            cd ~/recoup-agent-apis
+
+            # Initialize git repository if it doesn't exist
+            if [ ! -d .git ]; then
+              git init
+              git remote add origin https://github.com/sweetmantech/Recoup-Agent-APIs.git
+            fi
+
+            # Fetch latest code and handle local changes
+            git fetch --all
+            git checkout -B ${{ github.ref_name }} origin/${{ github.ref_name }}
+            git reset HEAD --hard
+            git pull origin/${{ github.ref_name }}
+
+            echo "Current branch:"
+            git branch --show-current
+            echo "Git status:"
+            git status
+
+            # Install bun globally if not already installed
+            if ! command -v bun &> /dev/null; then
+              echo "Installing bun globally..."
+              sudo curl -fsSL https://bun.sh/install | sudo bash
+              sudo ln -s ~/.bun/bin/bun /usr/local/bin/bun
+            fi
+
+            # Install pnpm if not already installed
+            if ! command -v pnpm &> /dev/null; then
+              echo "Installing pnpm globally..."
+              npm install -g pnpm
+            fi
+
+            # Ensure correct Node.js version
+            source ~/.nvm/nvm.sh
+            nvm use 20
+
+            # Install dependencies
+            pnpm install
+
+            # Restart the application
+            pm2 restart ecosystem.config.cjs --update-env

.github/workflows/test.yaml

@@ -0,0 +1,27 @@
+name: Test
+
+on:
+  pull_request:
+    branches:
+      - "main"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: "20"
+
+      - name: Install pnpm
+        run: npm install -g pnpm
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Run tests
+        run: pnpm test

.gitignore

@@ -0,0 +1,9 @@
+node_modules/
+.env
+package-lock.json
+dist/
+instructions.md
+ROADMAP.md
+# Cookie files
+**/*cookies.json
+scratchpad.md

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 @sweetman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

README.md

@@ -0,0 +1,53 @@
+# Recoup Agent APIs
+
+## Overview
+
+The Recoup Agent APIs provide a comprehensive interface for managing and interacting with the wrapped funnel system. This project aims to streamline the process of handling API interactions in a reliable and efficient manner.
+
+## Getting Started
+
+To get started with the Recoup Agent APIs:
+
+1. Clone the repository
+
+```bash
+git clone https://github.com/voicefirstai/Recoup-Agent-APIs.git
+cd Recoup-Agent-APIs
+```
+
+2. Install pnpm (if not already installed)
+
+```bash
+npm install -g pnpm
+```
+
+3. Install dependencies
+
+```bash
+pnpm install
+```
+
+4. Configure your environment
+
+```bash
+# Copy the example environment file
+cp .env.example .env
+
+# Edit the .env file with your configuration
+```
+
+5. Start the development server
+
+```bash
+pnpm start
+```
+
+6. Build for production
+
+```bash
+pnpm build
+```
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.