Updating of Neo4j DB version results in Python interface issues in API

[james-skelton]

The API currently uses the Neo4j Docker 3.5.34 container. When this was updated to 4.3.6, the API routes to interface with Neo4j failed. I will investigate this further with a local example and update findings here.

The reason that py2neo was selected over the neo4j driver for the API was because the neo4j driver did not have a way to return the data for edges as JSON. When edges were returned, the available data was superficial and did not have edge attributes.

[james-skelton]

So far, I've not been able to replicate the errors. Strategically, I will try:

Python version	Neo4j version	py2neo version	Create worked?	Match worked?
3.9.7	4.3.6	2021.2.3	Yes	Yes
3.9.7	4.4.8	2021.2.3	Yes	Yes

[james-skelton]

Perhaps the issue is with the data upload method (i.e., attempting to query after doing a neo4j-admin upload).

[james-skelton]

Managed to get a raised error by:

• Setting up a neo4j 4.4.8 container
• Importing gene, protein, and protein-encodes-gene data
• Restarting the container (which makes the imported data available)
• Using py2neo to run a query.

This resulted in a ConnectionBroken: Failed to read message error.

---------------------------------------------------------------------------
BrokenWireError                           Traceback (most recent call last)
File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:156, in BoltMessageReader.read_message(self)
    155 try:
--> 156     hi, lo = self.wire.read(2)
    157 except WireError as error:

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/wiring.py:128, in Wire.read(self, n)
    127         else:
--> 128             self.__mark_broken("Network read incomplete "
    129                                "(received %d of %d bytes)" %
    130                                (self.__input_len, n))
    131 data = self.__input[:n]

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/wiring.py:220, in Wire.__mark_broken(self, message)
    219 self.__broken = True
--> 220 raise BrokenWireError(message, idle_time=idle_time,
    221                       bytes_sent=self.__bytes_sent,
    222                       bytes_received=self.__bytes_received)

BrokenWireError: Network read incomplete (received 0 of 2 bytes) after 0.0s idle (258 bytes sent, 111164 bytes received)

The above exception was the direct cause of the following exception:

ConnectionBroken                          Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 for i in g.run("MATCH (n: Protein) RETURN n LIMIT 100"):    print(i)

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/database.py:405, in Graph.run(self, cypher, parameters, **kwparameters)
    395 def run(self, cypher, parameters=None, **kwparameters):
    396     """ Run a single read/write query within an auto-commit
    397     :class:`~py2neo.Transaction`.
    398 
   (...)
    403     :return:
    404     """
--> 405     return self.auto().run(cypher, parameters, **kwparameters)

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/database.py:992, in Transaction.run(self, cypher, parameters, **kwparameters)
    988     else:
    989         result = self._connector.auto_run(cypher, parameters,
    990                                           graph_name=self.graph.name,
    991                                           readonly=self.readonly)
--> 992     self._connector.pull(result, -1)
    993     return Cursor(result, hydrant)
    994 finally:

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/__init__.py:1434, in Connector.pull(self, result, n)
   1432 cx = self._reacquire(result.transaction)
   1433 try:
-> 1434     cx.pull(result, n=n)
   1435 except (ConnectionUnavailable, ConnectionBroken):
   1436     self.prune(cx.profile)

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:995, in Bolt4x0.pull(self, result, n, capacity)
    993 result.append(response, final=(n == -1))
    994 try:
--> 995     self._sync(response)
    996 except BrokenWireError as error:
    997     result.transaction.mark_broken()

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:799, in Bolt1._sync(self, *responses)
    797 self.send()
    798 for response in responses:
--> 799     self._wait(response)

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:794, in Bolt1._wait(self, response)
    787 """ Read all incoming responses up to and including a
    788 particular response.
    789 
    790 This method calls fetch, but does not raise an exception on
    791 FAILURE.
    792 """
    793 while not response.full() and not response.done():
--> 794     self._fetch()

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:769, in Bolt1._fetch(self)
    760 def _fetch(self):
    761     """ Fetch and process the next incoming message.
    762 
    763     This method does not raise an exception on receipt of a
   (...)
    767     failed state into an exception.
    768     """
--> 769     tag, fields = self.read_message()
    770     if tag == 0x70:
    771         self._responses.popleft().set_success(**fields[0])

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:696, in Bolt1.read_message(self)
    695 def read_message(self):
--> 696     tag, fields = self._reader.read_message()
    697     if tag == 0x71:
    698         # If a RECORD is received, check for more records
    699         # in the buffer immediately following, and log and
    700         # add them all at the same time
    701         while self._reader.peek_message() == 0x71:

File /opt/conda/miniconda3/envs/nedrex/lib/python3.9/site-packages/py2neo/client/bolt.py:158, in BoltMessageReader.read_message(self)
    156     hi, lo = self.wire.read(2)
    157 except WireError as error:
--> 158     raise_from(ConnectionBroken("Failed to read message"), error)
    159 else:
    160     if hi == lo == 0:
    161         # We hit a zero chunk...

File <string>:3, in raise_from(value, from_value)

ConnectionBroken: Failed to read message

Notably, this error only occurs on the first query I ran (subsequent queries succeeded) and may indicate something about how py2neo is expecting Neo4j to return the data.

Edit: This could also reflect a query being sent before the Neo4j server has restarted fully.