PropertyGraph `test_property_graph_mg.py::test_add_edge_data_with_ids` seg faults when restricted to a single-GPU

[rlratzel]

This is still light on details while being investigated more, but for now when the test is run using:

$> CUDA_VISIBLE_DEVICES=0 pytest -svx ./python/cugraph/cugraph/tests/data_store/test_property_graph_mg.py

A seg fault happens which produces the following stack trace:

python/cugraph/cugraph/tests/data_store/test_property_graph_mg.py::test_add_edge_data_with_ids[True] 2023-05-31 03:02:54,239 - distributed.preloading - INFO - Creating preload: dask_cuda.
initialize
2023-05-31 03:02:54,239 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

Dask client/cluster created using LocalCUDACluster
[ce950e66fa0c:104  :0:136] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x1bd8)
==== backtrace (tid:    136) ====
 0  /usr/local/lib/libucs.so.0(ucs_handle_error+0x2dc) [0x7f6a70899c1c]
 1  /usr/local/lib/libucs.so.0(+0x30dff) [0x7f6a70899dff]
 2  /usr/local/lib/libucs.so.0(+0x31134) [0x7f6a7089a134]
 3  /usr/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6aff641420]
 4  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x358d40) [0x7f6a2cc0bd40]
 5  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(__ptx37960+0x29) [0x7f6a2cc0ca49]
 6  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x3639f7) [0x7f6a2cc169f7]
 7  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x3639f7) [0x7f6a2cc169f7]
 8  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x3639f7) [0x7f6a2cc169f7]
 9  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x3639f7) [0x7f6a2cc169f7]
10  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x359d14) [0x7f6a2cc0cd14]
11  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x359d50) [0x7f6a2cc0cd50]
12  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(__ptx37934+0x8e) [0x7f6a2cc0a7be]
13  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(__ptx37958+0xb6) [0x7f6a2cc0d456]
14  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(__cuda_CallJitEntryPoint+0x10ad) [0x7f6a2cc0faed]
15  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(nvLinkerDestroy+0x15d) [0x7f6a2cbf218d]
16  /opt/conda/lib/python3.9/site-packages/cubinlinker/_nvlinklib.cpython-39-x86_64-linux-gnu.so(+0x33e885) [0x7f6a2cbf1885]
17  /opt/conda/bin/python3.9(+0x149df0) [0x557cb21bddf0]
18  /opt/conda/bin/python3.9(PyObject_Call+0x158) [0x557cb21bc1a8]
19  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x5ba6) [0x557cb21a2d36]
20  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
21  /opt/conda/bin/python3.9(_PyObject_FastCallDictTstate+0x13e) [0x557cb21a5b1e]
22  /opt/conda/bin/python3.9(_PyObject_Call_Prepend+0x69) [0x557cb21b9209]
23  /opt/conda/bin/python3.9(+0x2110d9) [0x557cb22850d9]
24  /opt/conda/bin/python3.9(_PyObject_MakeTpCall+0x2fc) [0x557cb21a62bc]
25  /opt/conda/bin/python3.9(+0x1da59f) [0x557cb224e59f]
26  /opt/conda/bin/python3.9(+0x1ddedf) [0x557cb2251edf]
27  /opt/conda/bin/python3.9(+0x14c05c) [0x557cb21c005c]
28  /opt/conda/bin/python3.9(+0x1259e6) [0x557cb21999e6]
29  /opt/conda/bin/python3.9(+0x14bc74) [0x557cb21bfc74]
30  /opt/conda/bin/python3.9(+0x131ec8) [0x557cb21a5ec8]
31  /opt/conda/bin/python3.9(+0x1282b0) [0x557cb219c2b0]
32  /opt/conda/bin/python3.9(+0x147935) [0x557cb21bb935]
33  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x123b) [0x557cb219e3cb]
34  /opt/conda/bin/python3.9(+0x139ef3) [0x557cb21adef3]
35  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x68b) [0x557cb219d81b]
36  /opt/conda/bin/python3.9(+0x139ef3) [0x557cb21adef3]
37  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x68b) [0x557cb219d81b]
38  /opt/conda/bin/python3.9(+0x139ef3) [0x557cb21adef3]
39  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x68b) [0x557cb219d81b]
40  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
41  /opt/conda/bin/python3.9(_PyFunction_Vectorcall+0xd5) [0x557cb21adc05]
42  /opt/conda/bin/python3.9(+0x147a16) [0x557cb21bba16]
43  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x402b) [0x557cb21a11bb]
44  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
45  /opt/conda/bin/python3.9(_PyObject_FastCallDictTstate+0x13e) [0x557cb21a5b1e]
46  /opt/conda/bin/python3.9(_PyObject_Call_Prepend+0x69) [0x557cb21b9209]
47  /opt/conda/bin/python3.9(+0x2110d9) [0x557cb22850d9]
48  /opt/conda/bin/python3.9(_PyObject_MakeTpCall+0x2fc) [0x557cb21a62bc]
49  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x4d62) [0x557cb21a1ef2]
50  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
51  /opt/conda/bin/python3.9(_PyFunction_Vectorcall+0xd5) [0x557cb21adc05]
52  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x123b) [0x557cb219e3cb]
53  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
54  /opt/conda/bin/python3.9(_PyFunction_Vectorcall+0xd5) [0x557cb21adc05]
55  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x123b) [0x557cb219e3cb]
56  /opt/conda/bin/python3.9(+0x139ef3) [0x557cb21adef3]
57  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x402b) [0x557cb21a11bb]
58  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
59  /opt/conda/bin/python3.9(_PyFunction_Vectorcall+0xd5) [0x557cb21adc05]
60  /opt/conda/bin/python3.9(_PyEval_EvalFrameDefault+0x68b) [0x557cb219d81b]
61  /opt/conda/bin/python3.9(+0x12827a) [0x557cb219c27a]
=================================
2023-05-31 03:03:13,384 - distributed.nanny - WARNING - Restarting worker
2023-05-31 03:03:14,600 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2023-05-31 03:03:14,600 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2023-05-31 03:03:17,364 - distributed.worker - WARNING - Compute Failed
Key:       ('join-indexed-3730f9db80b2f48b498c8c2119a7c40d', 1)
Function:  merge_chunk
args:      (           volume          time  card_num card_type  _SRC_  _DST_        _TYPE_
_EDGE_ID_
12           72.0  1.639085e+09      4321     DEBIT  78634     16  transactions, Empty DataFrame
Columns: [relationship_type, _SRC_, _DST_, _TYPE_]
Index: [])
kwargs:    {'how': 'outer', 'suffixes': ('', '_NEW_'), 'indicator': False, 'left_on': None, 'right_on': None, 'left_index': True, 'right_index': True, 'result_meta':            volume  time  card_num card_type  _SRC_  _DST_        _TYPE_  relationship_type  _SRC__NEW_  _DST__NEW_   _TYPE__NEW_
_EDGE_ID_
0             0.0   0.0         0       cat      0      0  transactions                  0           0           0  transactions
1             1.0   1.0         1       dog      1      1  transactions                  1           1           1  transactions}
Exception: 'TypeError("No dispatch for <class \'cudf.core.series.Series\'>")'

2023-05-31 03:03:17,411 - distributed.worker - WARNING - Compute Failed
Key:       ('join-indexed-3730f9db80b2f48b498c8c2119a7c40d', 2)
Function:  merge_chunk
args:      (           volume          time  card_num card_type  _SRC_  _DST_        _TYPE_
_EDGE_ID_
13          103.2  1.639085e+09     98124         V  32431      4  transactions, Empty DataFrame
Columns: [relationship_type, _SRC_, _DST_, _TYPE_]
Index: [])
kwargs:    {'how': 'outer', 'suffixes': ('', '_NEW_'), 'indicator': False, 'left_on': None, 'right_on': None, 'left_index': True, 'right_index': True, 'result_meta':            volume  time  card_num card_type  _SRC_  _DST_        _TYPE_  relationship_type  _SRC__NEW_  _DST__NEW_   _TYPE__NEW_
_EDGE_ID_
0             0.0   0.0         0       cat      0      0  transactions                  0           0           0  transactions
1             1.0   1.0         1       dog      1      1  transactions                  1           1           1  transactions}
Exception: 'TypeError("No dispatch for <class \'cudf.core.series.Series\'>")'

FAILED2023-05-31 03:03:47,792 - distributed.scheduler - ERROR - broadcast to tcp://127.0.0.1:32877 failed: OSError: Timed out trying to connect to tcp://127.0.0.1:32877 after 30 s
distributed.comm.core.CommClosedError: in <distributed.comm.tcp.TCPConnector object at 0x7fafa8b9c790>: ConnectionRefusedError: [Errno 111] Connection refused