Add Crawler Rate Limit #191
Description
It is beneficial to sometimes have rate limits that a Crawler should run with so it doesn't overwhelm targets and/or complies with robots limits. (from discord user jeremygiberson)
{{ message }}
It is beneficial to sometimes have rate limits that a Crawler should run with so it doesn't overwhelm targets and/or complies with robots limits. (from discord user jeremygiberson)