Fix RemoteDisconnected: Remote end closed connection without response #11

[thomasyu888]

This issue: https://github.com/nlpsandbox/data-node-client/issues/11, should be filed here instead of the data-node-client. Filing for @tschaffter

The following error occurs some time after starting uploading data with the script push_dataset.py. I first encountered this issue when pushing to a local data node and "solved" the issue by adding a sleep(1) after pushing the data for each patient. The same error then reappeared when pushing to our production data node running on an EC2. I tried the script locally and reported that it works fine for him without the sleep calls.

         '\n'
         '\n'}
Traceback (most recent call last):
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "push_dataset.py", line 131, in <module>
    annotation = annotation_api.create_annotation(
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/api/annotation_api.py", line 65, in create_annotation
    return self.create_annotation_with_http_info(dataset_id, annotation_store_id, **kwargs)  # noqa: E501
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/api/annotation_api.py", line 156, in create_annotation_with_http_info
    return self.api_client.call_api(
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/api_client.py", line 365, in call_api
    return self.__call_api(resource_path, method,
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/api_client.py", line 182, in __call_api
    response_data = self.request(
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/api_client.py", line 408, in request
    return self.rest_client.POST(url,
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/rest.py", line 266, in POST
    return self.request("POST", url,
  File "/mnt/c/Users/thoma/Documents/dev/nlpsandbox/data-node-client/datanodeclient/rest.py", line 164, in request
    r = self.pool_manager.request(
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/util/retry.py", line 531, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/home/tschaffter/.conda/envs/data-node-client/lib/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Observations

• I repeated the same script 10 times using the production server and the behavior seems random but at least 3 times the script showed the same clinical note just before throwing the error.
• I repeated the same script 10 times using the local server and I had not issues pushing the entire dataset.

[tschaffter]

Tried using --http instead of --http-socket. The error still happens sometimes.

for run in {1..20}; do python push_dataset.py || sleep 5; done

[tschaffter]

Adding http-keepalive = 1 seems to solve the issue. No error in 20 runs.

[tschaffter]

TODO

Document the use of command: uwsgi --ini uwsgi.ini --lazy --http :8080 --processes 4 if this helps us to fix the issue with the data node.

[tschaffter]

There is no magic rule for setting the number of processes or threads to use. It is very much application and system dependent. Simple math like processes = 2 * cpucores will not be enough. You need to experiment with various setups and be prepared to constantly monitor your apps. uwsgitop could be a great tool to find the best values.