Unhelpful hdfs.util.HdfsError: b'' on HTTP/1.1 500 Server Error response #211
Description
The server responds to a PUT request with an error 500 (this turned out to be because I was using an unsupported Java version to run HDFS, i.e. wrong JAVA_HOME):
[pid 2801510] connect(10, {sa_family=AF_INET, sin_port=htons(9870), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
[pid 2801510] sendto(10, "PUT /webhdfs/v1/test/test.parquet?user.name=chris&overwrite=True&op=CREATE HTTP/1.1\r\nHost: localhost:9870\r\nUser-Agent: python-re"..., 273, 0, NULL, 0) = 273
[pid 2801510] ioctl(10, FIONBIO, [0]) = 0
[pid 2801510] recvfrom(10, "HTTP/1.1 500 Server Error\r\nConnection: close\r\n\r\n", 8192, 0, NULL, NULL) = 48
[pid 2801510] recvfrom(10, "", 8192, 0, NULL, NULL) = 0
[pid 2801510] close(10) = 0
[pid 2801510] newfstatat(AT_FDCWD, "/usr/lib/python3.10/cmd.py", {st_mode=S_IFREG|0644, st_size=14860, ...}, 0) = 0
[pid 2801510] write(1, "*** hdfs.util.HdfsError: b''\n", 29) = 29
But all the user sees is hdfs.util.HdfsError: b'', which is very unhelpful (especially if you don't have access to the NameNode logs). The traceback is:
with self.client.write(self._path(name), overwrite=True) as writer:
File "/home/chris/.../.venv/lib/python3.10/site-packages/hdfs/client.py", line 497, in write
res = self._create(
File "/home/chris/.../.venv/lib/python3.10/site-packages/hdfs/client.py", line 118, in api_handler
raise err
Probably because _to_error does not handle this status code very well. There is not much info in the exception:
(Pdb) type(response)
<class 'requests.models.Response'>
(Pdb) p response.status_code
500
(Pdb) p response.text
''
(Pdb) p response.content
b''
(Pdb) p response.headers
{'Connection': 'close'}
(Pdb) p response.raw.read()
b''
(Pdb) p response.reason
'Server Error'
Maybe the best we can do is raise_for_status?
(Pdb) p str(response.raise_for_status())
*** requests.exceptions.HTTPError: 500 Server Error: Server Error for url: http://localhost:9870/webhdfs/v1/test/test.parquet?user.name=chris&overwrite=True&op=CREATE