Add detection of standard URL extensions to HTTP `is_dir` and increase efficiency of fancy indexing

When checking for a directory, first look at the URL and return `False` if the URL ends with any standard HTML extensions. See Apache extension list.

Also we could look for the "Parent directory" fancy index marker earlier in the file download process rather than reading the maximum amount from the URL first.
