Fatal error with http driver at Python exit

[LarsKoeppel]

Hi,

I am currently trying to integrate tensorstore into knossos_utils.
For the most part, it works fine. However, when I load a dataset with the HTTP driver, there is a Python crash at the end when Python exits (trace at the end). With the file driver, this does not happen.

It can be replicated with the following code. I am using Python 3.12 with the currently available version of tensorstore (0.1.80) from pip.

from knossos_utils import KnossosDataset
kd = KnossosDataset("https://data.ariadne.ai/public/cuber_test/7s8801o5v0n798ihe8ny3ffbklvwktep/test.streaming.k.toml")

In the library, multiple handlers are opened and stored in a dict to have direct access to each downsampled version (Link to code)
Is it a known limitation that you cannot open multiple handlers on the same dataset?

I would like to keep the handlers open to reduce the overhead while working with the data. However, if this is not possible, I will look into options to open them only on demand and close them after the read/write is done.

Any help is appreciated!
Best regards

Trace out of Dr.Mingw:

python.exe caused a Fast Fail at location 00007FFAE5934ACE in module ucrtbase.dll with code 7 (FATAL_APP_EXIT).

AddrPC           Params
00007FFAE5934ACE 0000007E00000003 0000007E00000003 0000007E81BEE7C0  ucrtbase.dll!abort+0x4e
00007FFAE59519DE 0000007E81BEF160 0000007E81BEE0B0 0000007E81BEE7C0  ucrtbase.dll!terminate+0x1e
00007FFAC8AE1D8D 0000000000000005 00007FFAC8AE55D2 00007FFACB310000  VCRUNTIME140.dll!FindHandler<__FrameHandler3>+0x3b9  [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 780]
00007FFAC8AE2D90 00007FFACB310000 0000007E81BEF160 0000007E81BEE9A0  VCRUNTIME140.dll!__InternalCxxFrameHandler<__FrameHandler3>+0x208  [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\frame.cpp @ 396]
00007FFAC8AEF541 0000007E81BEF3B0 0000007E81BEF160 0000007E81BEE770  VCRUNTIME140.dll!__CxxFrameHandler3+0x71  [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\risctrnsctrl.cpp @ 281]
00007FFAE87062FF 0000007E81BEF160 00007FFA21CD0000 00007FFA2261239C  ntdll.dll!RtlpExecuteHandlerForException+0xf
00007FFAE85B2327 0000000000000000 0000007E81BEF010 0000000000000000  ntdll.dll!RtlDispatchException+0x437
00007FFAE85AA961 0000007E81BEF2C0 0000007E81BEF268 0000000000000001  ntdll.dll!RtlRaiseException+0x221
00007FFAE600782A 0000007E0000000B 00007FFACB38F2F0 0000019C7A6EDCC0  KERNELBASE.dll!RaiseException+0x8a
00007FFAC8AE5267 0000019C7A6EDBD0 0000000000000006 0000000000000006  VCRUNTIME140.dll!_CxxThrowException+0x97  [D:\a\_work\1\s\src\vctools\crt\vcruntime\src\eh\throw.cpp @ 81]
00007FFACB323246 0000007E81BEF3F0 0000007E81BEF420 0000000000000000  MSVCP140.dll!std::_Throw_Cpp_error+0x46  [D:\a\_work\1\s\src\vctools\crt\github\stl\src\thread0.cpp @ 33]
00007FFA23276AE6 00007FFA233D14FC 0000000000000001 0000019C7A0A5A60  _tensorstore.cp312-win_amd64.pyd!PyInit__tensorstore+0x13e6c16
00007FFA2261239C 0000000000000000 0000019C7AE83CE0 0000F5BE7EE0DA42  _tensorstore.cp312-win_amd64.pyd!PyInit__tensorstore+0x7824cc
00007FFA2343EB80 0000019C7A0A5A60 0000000000000040 FFFFFFFFFFFFFFFF  _tensorstore.cp312-win_amd64.pyd!PyInit__tensorstore+0x15aecb0
00007FFAE58DBC75 0000007E81BEF500 0000007E81BEF528 0000007E81BEF810  ucrtbase.dll!<lambda_f03950bc5685219e0bcd2087efbe011e>::operator()+0xa5
00007FFAE58DB897 0000000000000000 0000000000000001 00007FFACE79FDC0  ucrtbase.dll!__crt_seh_guarded_call<int>::operator()<<lambda_7777bce6b2f8c936911f934f8298dc43>,<lambda_f03950bc5685219e0bcd2087efbe011e> &,<lambda_3883c3dff614d5e0c5f61bb1ac94921c> >+0x3b
00007FFAE58DB84D 00007FFA23CEA700 0000000000000002 0000FB9200000002  ucrtbase.dll!_execute_onexit_table+0x3d
00007FFA233D1395 0000000000000001 00007FFAE8717F90 0000000000000000  _tensorstore.cp312-win_amd64.pyd!PyInit__tensorstore+0x15414c5
00007FFA233D14B5 00007FFA21CD0000 0000019C00000000 0000000000000001  _tensorstore.cp312-win_amd64.pyd!PyInit__tensorstore+0x15415e5
00007FFAE86FF86E 0000000000000000 0000000000000000 0000000000000000  ntdll.dll!LdrpCallInitRoutineInternal+0x22
00007FFAE85ABCAE 0000019C7A5EE0E0 00007FFA21CD0000 00007FFA00000000  ntdll.dll!LdrpCallInitRoutine+0x10e
00007FFAE862D4AF 0000000000000000 0000000000000000 0000019C57D07110  ntdll.dll!LdrShutdownProcess+0x17f
00007FFAE862C67E 0000000000000000 0000000000000000 FFFFFFFFFFFFFFFE  ntdll.dll!RtlExitUserProcess+0x9e
00007FFAE6CB18AB 0000000000000000 0000000000000000 0000000000000000  KERNEL32.DLL!ExitProcessImplementation+0xb
00007FFAE5930093 0000000000000000 0000000000000000 0000000000000000  ucrtbase.dll!common_exit+0xc7
00007FF6A5241297 0000000000000000 0000000000000000 0000000000000000  python.exe!0x1297
00007FFAE6C9E8D7 0000000000000000 0000000000000000 000004F0FFFFFB30  KERNEL32.DLL!BaseThreadInitThunk+0x17
00007FFAE862C53C 0000000000000000 0000000000000000 0000000000000000  ntdll.dll!RtlUserThreadStart+0x2c

[laramiel]

Can you isolate this to a smaller test case? Shutdown can have ordering issues, and if there were a self-contained reproducer then I could better debug it.

[LarsKoeppel]

I will try to create a smaller test case tomorrow.

[LarsKoeppel]

Here is a minimal excample that replicates the error. It is basicaly just opening a dataset with the http driver.
I am using python 3.12.10 under Windows. The script is started in VS Code with the normal debugger.

import urllib
import tensorstore as ts
import requests

url = "https://data.ariadne.ai/public/cuber_test/7s8801o5v0n798ihe8ny3ffbklvwktep/data/"

response = requests.get(url + "info")
response.raise_for_status()
info_json = response.json()

keys = [scale["key"] for scale in info_json["scales"]]

parsed_url = urllib.parse.urlparse(url)
driver = "http"
key = keys[0]
data_path = parsed_url.path
dataset = ts.open({
                "driver": "neuroglancer_precomputed",
                "kvstore": {
                    "base_url": f"{parsed_url.scheme}://{parsed_url.hostname}",
                    "driver": driver,
                    "path": data_path,
                },
                # "scale_metadata" : {
                #     "key" : key,
                # },
            }).result()