Rendered documentation for GitHub Pages

[maelle]

I've just seen

urlchecker-python/urlchecker/core/urlproc.py

Line 182 in 7fd0886

# Some sites will return 403 if it's not a "human" user agent

and this is brilliant (the user agent setting)! What about listing the strengths of this library in the README?

[vsoch]

We could do that! Although I'm not sure pretending to be an actual user agent (and not a bot) is something that I'd want to show case, some people might not be happy about it. What do you think @SuperKogito ?

[SuperKogito]

So I did not knew about user agents before and when I saw them for the first time, I was very impressed. I agree to @maelle point, the user agents option can be attractive to many users as it simulates a user behaviour, which makes the tests more realistic.

On the other hand, to @vsoch point: I think many websites won't appreciate the extra/unwanted deceptive traffic simulating users as this would make their statistics of Web browser usage inaccurate (User agent spoofing) and excluding bots won't disable the checks we run.

This seems to be a grey area so from what I see, we have 2 options:

• Option 1: provide transparent documentation as this is a testing tool and users should bare the responsibility of any miss-use, but since we want to run continuous analytics, this can comes back to bite us :/

• Option 2: not include this for now, and see how this develops in the future. Just wait and see, maybe do more research on the general practises for such tools.

As much as I want to go with the first option (since I am all for transparent good docs), I think we should go with option 2, at least for now.

@vsoch can you elaborate on what can go wrong here in your opinion? is it only what I mentioned or is there more to it?

[vsoch]

It's not super terrible, there are plenty of posts (e.g., this one) that talk about doing the same thing. I personally don't see the need to have bullets in documentaiton or a readme that say "This library is so great because" for functionality that isn't relevant for the user. For example, providing example recipes in urlchecker-action is important - the user needs to write a recipe. The user doesn't care about the user agent strings, they just need the tests to work. This is why I think it should be left out, not for some reason that we are doing something totally off base.

[SuperKogito]

I agree. I would say most users will be interested in the library as whole (How to use it? Does it fulfil the task?) rather than its mechanics.

[vsoch]

So @maelle thank you for the suggestion - let's rescope to just overall improving documentation. We just have a github readme thus far, and at some point we can render something pretty instead.