Clarifications regarding crawler IP ranges

[sebastian-nagel]

crawler operators should expose the IP ranges they allocated for crawling in a standardized, machine readable format, and keep it reasonably up to date (i.e. shouldn't get older than 7 days).

1. Replace "shouldn't get older than 7 days" by "shouldn't get outdated for more than 7 days"?
   The term "old" is unspecific: it could also mean that the file is required to be touched every 7 days, without keeping the information up-to-date.

2. Is a 7-day update period sufficient?
   There are at least the following use cases for the list of IP address ranges:

   1. verify requests in web server access logs.
   2. configure IP blocking or explicitly allowing the specified IP ranges.
   3. IP targeting which includes also "black hat" techniques such as "cloaking".

   For (i), "historic" IP ranges are a requirement, otherwise verification may raise false alarms about faked user-agent strings in past requests. This would also recommend to attach time periods to IP address ranges.

   For (ii) and (iii), a 7-day update period seems overtly long. New IP ranges should be announced even ahead of time. Since we have to assume that everybody plays nice, "black hat" techniques are no counter argument for announcing IP address ahead of time. However, they might be mentioned in the section "Security Considerations".

3. An example JSON schema would help to keep variations of crawlerips.json at a minimum. At least, it should be mentioned what "standardized" means – "constant" for the crawler with a published description / specification or whether it is part of a globally defined standard (maybe upcoming).

4. What about reverse DNS as an alternative or augmenting technique for crawler verification? Should this be mentioned or even added as a section?

[garyillyes]

Howdy!

I mentioned this in the interim meeting, but to put it here as well: I won't be able to develop this I-D further for various reasons. I would LOVE if someone could pick it up and develop it further though together with publishers, and then bring it back to the IETF (not yet sure which WG) for adoption.

[garyillyes]

Looks like we're gonna work on this. Feel free to look at the other open issues (specifically #4) and, if you feel like it, send PRs