pybgpstream not able to handle HTTP to HTTPS redirections

[dacCR]

Hi,

We've been trying to find a way around this for a while now but cannot find one.
Searching the Internet has not come up with anything that helps.

Wondering how others are handling this problem of the http links being redirected to https.

[dacCR]

I can't believe I'm the only one experiencing this. Is there a work around?
Where should I post to try and get help and simply report this if it's not well known if this is not the place?

[salcock]

Hi there,

Can you provide an example of a situation where a HTTPS redirect fails?

[dacCR]

Hi,

Sure,

RuntimeError: Could not get next record (is the stream started?)
1704914803 HTTP ERROR: SSL connect error (35)
2024-01-10 19:26:43 111915: bs_transport_file.c:39: ERROR: Could not open http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz for reading
2024-01-10 19:26:43 111915: bgpstream_transport.c:97: ERROR: Could not open resource (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz)
2024-01-10 19:26:43 111915: bgpstream_reader.c:169: WARNING: Could not open (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz). Attempt 5 of 5
2024-01-10 19:26:43 111915: bgpstream_reader.c:181: ERROR: Could not open dumpfile (http://data.ris.ripe.net/rrc21/2024.01/updates.20240109.1435.gz) after 5 attempts. Giving up

Based on tests with wget and curl, it's clear that the URLs are being redirected from HTTP to HTTPS. However, while wget automatically follows the redirect and successfully downloads the file, curl does not follow the redirect by default and instead downloads the HTML content of the 301 Moved Permanently page.

In the case of my Python script, it appears that pybgpstream might not be handling the HTTP to HTTPS redirection properly, leading to the SSL connection errors.

[hanskuhn]

I'm just lurking, but glad to see this issue raised.

RouteViews data archive (archive.routeviews.org) does not currently issue a redirect to HTTPS, but it's on my wishlist to implement. I've been hesitating because I had a suspicion that lots of tooling for BGP data will need to be updated. 👀

[dacCR]

I don't know if this helps or not but I decided to try to make sure everything was up to date.
It may not be an issue but I'm taking off for several days in a few minutes so will look at this when I get back.
However, I thought I would share just in case it's related.

pip freeze > requirements.txt 
pip install -r requirements.txt --upgrade 

root@bgp01:/new# pip freeze > requirements.txt
root@bgp01:/new# pip install -r requirements.txt --upgrade
Requirement already satisfied: attrs==21.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (21.2.0)
Requirement already satisfied: Automat==20.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (20.2.0)
Requirement already satisfied: Babel==2.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 3)) (2.8.0)
Requirement already satisfied: bcrypt==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 4)) (3.2.0)
Requirement already satisfied: blinker==1.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (1.4)
Requirement already satisfied: certifi==2020.6.20 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2020.6.20)
Requirement already satisfied: chardet==4.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 7)) (4.0.0)
Requirement already satisfied: click==8.0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (8.0.3)
Requirement already satisfied: cloud-init==23.1.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 9)) (23.1.2)
Requirement already satisfied: colorama==0.4.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 10)) (0.4.4)
Requirement already satisfied: command-not-found==0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 11)) (0.3)
Requirement already satisfied: configobj==5.0.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 12)) (5.0.6)
Requirement already satisfied: constantly==15.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 13)) (15.1.0)
Requirement already satisfied: cryptography==3.4.8 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 14)) (3.4.8)
Requirement already satisfied: dbus-python==1.2.18 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 15)) (1.2.18)
Requirement already satisfied: distro==1.7.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 16)) (1.7.0)
Requirement already satisfied: distro-info===1.1build1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 17)) (1.1build1)
Requirement already satisfied: httplib2==0.20.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 18)) (0.20.2)
Requirement already satisfied: hyperlink==21.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 19)) (21.0.0)
Requirement already satisfied: idna==3.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 20)) (3.3)
Requirement already satisfied: importlib-metadata==4.6.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 21)) (4.6.4)
Requirement already satisfied: incremental==21.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 22)) (21.3.0)
Requirement already satisfied: jeepney==0.7.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 23)) (0.7.1)
Requirement already satisfied: Jinja2==3.0.3 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 24)) (3.0.3)
Requirement already satisfied: jsonpatch==1.32 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 25)) (1.32)
Requirement already satisfied: jsonpointer==2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 26)) (2.0)
Requirement already satisfied: jsonschema==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 27)) (3.2.0)
Requirement already satisfied: keyring==23.5.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 28)) (23.5.0)
Requirement already satisfied: launchpadlib==1.10.16 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 29)) (1.10.16)
Requirement already satisfied: lazr.restfulclient==0.14.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 30)) (0.14.4)
Requirement already satisfied: lazr.uri==1.0.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 31)) (1.0.6)
Requirement already satisfied: MarkupSafe==2.0.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 32)) (2.0.1)
Requirement already satisfied: more-itertools==8.10.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 33)) (8.10.0)
Requirement already satisfied: mysql-connector-python==8.2.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 34)) (8.2.0)
Requirement already satisfied: netifaces==0.11.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 35)) (0.11.0)
Requirement already satisfied: oauthlib==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 36)) (3.2.0)
Requirement already satisfied: pexpect==4.8.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 37)) (4.8.0)
Requirement already satisfied: protobuf==4.21.12 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 38)) (4.21.12)
Requirement already satisfied: ptyprocess==0.7.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 39)) (0.7.0)
Requirement already satisfied: pyasn1==0.4.8 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 40)) (0.4.8)
Requirement already satisfied: pyasn1-modules==0.2.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 41)) (0.2.1)
Requirement already satisfied: pybgpstream==2.0.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 42)) (2.0.2)
Requirement already satisfied: PyGObject==3.42.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 43)) (3.42.1)
Requirement already satisfied: PyHamcrest==2.0.2 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 44)) (2.0.2)
Requirement already satisfied: PyJWT==2.3.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 45)) (2.3.0)
Requirement already satisfied: pyOpenSSL==21.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 46)) (21.0.0)
Requirement already satisfied: pyparsing==2.4.7 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 47)) (2.4.7)
Requirement already satisfied: pyrsistent==0.18.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 48)) (0.18.1)
Requirement already satisfied: pyserial==3.5 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 49)) (3.5)
Requirement already satisfied: python-apt==2.4.0+ubuntu1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 50)) (2.4.0+ubuntu1)
Requirement already satisfied: python-dateutil==2.8.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 51)) (2.8.2)
Requirement already satisfied: python-debian===0.1.43ubuntu1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 52)) (0.1.43ubuntu1)
Requirement already satisfied: python-magic==0.4.24 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 53)) (0.4.24)
Requirement already satisfied: pytz==2022.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 54)) (2022.1)
Requirement already satisfied: PyYAML==5.4.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 55)) (5.4.1)
Requirement already satisfied: requests==2.25.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 56)) (2.25.1)
Requirement already satisfied: SecretStorage==3.3.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 57)) (3.3.1)
Requirement already satisfied: service-identity==18.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 58)) (18.1.0)
Requirement already satisfied: six==1.16.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 59)) (1.16.0)
Requirement already satisfied: sos==4.4 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 60)) (4.4)
Requirement already satisfied: ssh-import-id==5.11 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 61)) (5.11)
Requirement already satisfied: systemd-python==234 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 62)) (234)
Requirement already satisfied: Twisted==22.1.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 63)) (22.1.0)
Requirement already satisfied: ubuntu-advantage-tools==8001 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 64)) (8001)
Requirement already satisfied: ubuntu-drivers-common==0.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 65)) (0.0.0)
Requirement already satisfied: ufw==0.36.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 66)) (0.36.1)
Requirement already satisfied: unattended-upgrades==0.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 67)) (0.1)
Requirement already satisfied: urllib3==1.26.5 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 68)) (1.26.5)
Requirement already satisfied: wadllib==1.3.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 69)) (1.3.6)
Requirement already satisfied: xkit==0.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 70)) (0.0.0)
Requirement already satisfied: zipp==1.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 71)) (1.0.0)
Requirement already satisfied: zope.interface==5.4.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 72)) (5.4.0)
Collecting pycairo>=1.16.0
  Downloading pycairo-1.25.1.tar.gz (347 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 347.1/347.1 KB 1.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pycairo
  Building wheel for pycairo (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pycairo (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/cairo
      copying cairo/__init__.py -> build/lib.linux-x86_64-3.10/cairo
      copying cairo/__init__.pyi -> build/lib.linux-x86_64-3.10/cairo
      copying cairo/py.typed -> build/lib.linux-x86_64-3.10/cairo
      running build_ext
      'pkg-config' not found.
      Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycairo
Failed to build pycairo
ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects

[dacCR]

Does anyone know how to solve this? Updating everything didn't work.

[salcock]

This is still on my radar, but I have not had a lot of time to investigate the issue so far.

[dacCR]

I'm happy to run more tests for you if it'll help. Just not sure what those could be.
Anyone that's built a script and isn't watching closely would not even be aware their code is not working anymore.

[salcock]

Just a quick update -- I believe I've tracked down the problem and have a potential solution.

The underlying problem is within the libbgpstream code. Once I've completed my testing and confirmed that the problem is resolved, I'll submit a PR with the fix.

[dacCR]

Wonderful, thank you for the update.

[salcock]

A PR with the fix has now been submitted and is awaiting review: CAIDA/libbgpstream#245

[dacCR]

I'm not terribly familiar with github, can you let us know when it's approved and/or how to upgrade?

[dacCR]

I just checked the link you shared and it seems no one has reviewed it and it also says merge blocked.
I'm a little confused. Why would people use and count on this this if it won't get updated?

[dac2020]

Upgraded just fine and works, thank you.