Google Safe Browsing

[mta59066]

Not necessarily a pull request but a work in progress and discussion.

I added a mode option and trying to have the code successfully perform all normalization tests mentioned on the below document when run with 'safebrowsing' mode. So far 30 out of 34 tests are successful. Rationale comes from the Canonicalization section of this document https://developers.google.com/safe-browsing/developers_guide_v2

Not sure if the mode is the correct way to go. Some of the fixes introduced are specific to Google Safe Browsing rules, such as removing the fragment part of the query. Most people will expect the url to contain the fragment after normalization, but instead of a mode this might be an option when getting back the normalized url as a whole. Also converting a non ascii host to percent-escape or punycode or leaving as it is might need to be an option.

There are also other fixes which I left out of the 'safebrowsing' mode as they did not break tests with the original code.

[glenscott]

Hi,

Thanks for the interesting comments, I had not heard of the Google canonicalization process before.

I originally made the class as way of following the URL normalization rules specified in the RFC 3986. The canonicalization seems to follow many of these rules, but also adds some others.

Adding a mode is an interesting approach, but it leads to lots of if (mode==foo) in the codebase.

Perhaps subclassing the Normalizer class and overridding the methods that need Google-specific rules is the way to go?