Skip to content

feat: support scheduled ingestion for all source types (github, reddit, youtube, sessions) #46

@jmagar

Description

@jmagar

Problem

refresh schedule currently only works with url/urls (web pages). There's no way to schedule periodic re-ingestion for GitHub repos, Reddit feeds, YouTube channels, or session exports.

Current Behavior

// ingest has no schedule subaction
{ "action": "ingest", "subaction": "start", "source_type": "github", "target": "owner/repo" }

// refresh schedule only accepts url/urls
{ "action": "refresh", "subaction": "schedule", "url": "https://..." }
// unknown field `source_type` error if you try to pass it

Desired Behavior

Scheduling should work for any ingestion source type:

{ "action": "ingest", "subaction": "schedule", "source_type": "github", "target": "owner/repo", "interval": "24h" }
{ "action": "ingest", "subaction": "schedule", "source_type": "reddit", "target": "rust", "interval": "6h" }
{ "action": "ingest", "subaction": "schedule", "source_type": "youtube", "target": "https://youtube.com/...", "interval": "12h" }

Or alternatively, extend refresh schedule to accept source_type + target in addition to url:

{ "action": "refresh", "subaction": "schedule", "schedule_subaction": "create", "source_type": "github", "target": "owner/repo", "interval": "24h" }

Use Case

Keep a GitHub repo's indexed content fresh without manual re-ingestion. For example, a project repo that's actively developed should be re-indexed daily so queries reflect the latest code.

Suggested Approach

  • Add schedule subaction to ingest (mirrors the existing refresh schedule pattern)
  • Or generalize the refresh schedule storage model to store { source_type, target } pairs in addition to URL lists
  • Scheduled jobs should appear in ingest list / refresh schedule list with their next-run time

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions