Accessing live matches 403 error on Heroku

[rakemen]

My telegram bot is deployed on Heroku. I always get a 403 Forbidden error when I try to get live matches info or access any links on mapps.cricbuzz.com/cbzios/. When I try to run locally everything works fine. Could it be that all Heroku IP addresses are blocked by the cricbuzz website? Any luck deploying your code on Heroku?

[imaginarystarlord]

I am also getting the same problem

[rakemen]

I am also getting the same problem

Do you have any workarounds in mind?

[imaginarystarlord]

Yes

[rakemen]

@imaginarystarlord Can you share the idea, so that it may be helpful to others. Thanks!

[codophobia]

@rakemen Yes, it looks like ip address of the cloud servers are blocked. One way to get around this is to use a proxy address and then make a request. You can find many proxy ip addresses online.

export http_proxy=http://68.183.108.113:8080/
curl mapps.cricbuzz.com/cbzios/match/livematches

The above request works on azure and should also work on other cloud instances. So, the solution will be to set up "http_proxy" environment variable before using the library.

[imaginarystarlord]

we have to rotate IP with rotating user agent

[swainsandeep]

I have tried with the above ip in heroku,it is not working can anyone suggest any other ip,or where can I get those ips.

[swainsandeep]

@rakemen Yes, it looks like ip address of the cloud servers are blocked. One way to get around this is to use a proxy address and then make a request. You can find many proxy ip addresses online.

export http_proxy=http://68.183.108.113:8080/
curl mapps.cricbuzz.com/cbzios/match/livematches

The above request works on azure and should also work on other cloud instances. So, the solution will be to set up "http_proxy" environment variable before using the library.

@codophobia I have tried with the ip provided by you it is not working in heroku,can u suggest any other ip adress,can u guide me where I can get those ips.

[imaginarystarlord]

What u want to do?

[codophobia]

@swainsandeep https://free-proxy-list.net/

[swainsandeep]

@codophobia thanks for Sharing free proxy list.

I have tried one of the proxy in the provided site.Still same issue facing in Pythonanywhere site.

export http_proxy=http://51.38.71.101:8080

from pycricbuzz import Cricbuzz
c = Cricbuzz()
c.matches()

Getting this error:-

Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed to establish a new connection: [Errno 111] Con
nection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='51.38.71.101', port=8080): Max retries exceeded with url: http://mapps.cricbuzz.com/cbzios/matc
h/livematches (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed t
o establish a new connection: [Errno 111] Connection refused',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/pycricbuzz/cricbuzz.py", line 73, in matches
crawled_content = self.crawl_url(url)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/pycricbuzz/cricbuzz.py", line 10, in crawl_url
r = requests.get(url).json()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='51.38.71.101', port=8080): Max retries exceeded with url: http://mapps.cricbuzz.com/cbzios/match/
livematches (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed to
establish a new connection: [Errno 111] Connection refused',)))

[imaginarystarlord]

You have to rotate ips with rotating user agent, continuously, because these are free ips, they work sometimes, they may not work some times

[swainsandeep]

@imaginarystarlord imaginarystarlord Thanks for the reply.Can you share some code snippet for this.How to rotate ips using rotating user agent.

[swainsandeep]

@codophobia @imaginarystarlord Hi All,Anyone has any official document for cricbuzz for the usinng the api.Actually I raised this issue in pythonanywhere,they are asking for official documentation to to whitelist the api in Pythonanwhere.