Scheduling Crawl jobs? Like cron jobs #1800
-
|
Is there some capability to schedule Crawl jobs on the docker service, like cron jobs? E.g. Crawl this website every x minutes or once a day at 5pm? Or does this require a wrapper? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
From my point of view, scheduling feels like an orchestration concern rather than a crawling concern unless the product is intentionally becoming a persistent job runner. The crawler itself mainly needs idempotent invocation, clear job status, and a clean API boundary that an external scheduler can drive reliably. If those primitives are strong, cron-like behavior can stay outside the core without users losing anything important. |
Beta Was this translation helpful? Give feedback.
From my point of view, scheduling feels like an orchestration concern rather than a crawling concern unless the product is intentionally becoming a persistent job runner. The crawler itself mainly needs idempotent invocation, clear job status, and a clean API boundary that an external scheduler can drive reliably.
If those primitives are strong, cron-like behavior can stay outside the core without users losing anything important.