Use urljoin to fix relative urls #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

onilton wants to merge 1 commit into jmg:master from onilton:urljoin

onilton commented Jul 11, 2014

https://docs.python.org/2/library/urlparse.html#urlparse.urljoin provides a robust way to make a relative url into a absolute one.

This fixes some issues like this one:

When accessing this url:
http://www1.abracom.org.br/cms/opencms/abracom/pt/associados/

We find relative links like this:
resultado_busca.html?letra=a

The browser (chrome) build the absolute url like this:
http://www1.abracom.org.br/cms/opencms/abracom/pt/associados/resultado_busca.html?letra=a

But crawley build the url like this:
http://www1.abracom.org.br/resultado_busca.html?letra=a

urljoin fixes the issue, keeping the right behavior for /relativeurl:

In a hypothetical page http://mydomain.com/my/web/page.html:

'/relativeurl.html' link should become 'http://mydomain.com/relativeurl.html'

and

'relativeurl.html' link should become 'http://mydomain.com/my/web/relativeurl.html'


          Use urljoin to fix relative urls

4a47572

Author

onilton commented Nov 13, 2015

Any news on that? :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet