nsX files showing num_data_pts=0

[luiztauffer]

Describe the bug
I'm getting an OverflowError error when trying to read from ns2 and ns6 files from he same recording session. File Spec 2.3

To Reproduce

import brpylib

ns2 = brpylib.NsxFile("H4.3_openField_PPX_091423001.ns2")

ns2.basic_header    # works fine
ns2.extended_headers[-5:]    # works fine
d = ns2.getdata(elec_ids=[10])    # breaks here

error:

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
Cell In[5], line 1
----> 1 d = ns2.getdata(elec_ids=[1])

File ~/anaconda3/envs/env_nelson/lib/python3.13/site-packages/brpylib/brpylib.py:1204, in NsxFile.getdata(self, elec_ids, start_time_s, data_time_s, downsample, zeropad, full_timestamps, elec_rows, force_srate)
   1202 timestamp = unpack(ts_type, self.datafile.read(ts_size))[0]
   1203 num_data_pts = unpack("<I", self.datafile.read(4))[0]
-> 1204 timestamp = timestamp + (clk_per_samp * np.arange(num_data_pts)).astype(np.int64 if ts_size==8 else np.int32)
   1205 bod = self.datafile.tell()  # Beginning of segment data
   1206 output["data_headers"].append({
   1207     "Timestamp": timestamp,
   1208     "NumDataPoints": num_data_pts,
   1209     "data_time_s": num_data_pts / output["samp_per_s"]
   1210 })

OverflowError: Python integer 3439329535 out of bounds for int32

• OS: Linux (Ubuntu 20.04)
• Python 3.13.2
• Python-Utilities: main branch
• Numpy 2.2.4

[luiztauffer]

UPDATE
ok, the error above might have been fixed by downgrading numpy to 1.26.4

Now, I'm getting a ValueError error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 d = ns2.getdata(elec_ids=[1])

File /mnt/shared_storage/Github/Blackrock-Python-Utilities/brpylib/brpylib.py:1213, in NsxFile.getdata(self, elec_ids, start_time_s, data_time_s, downsample, zeropad, full_timestamps, elec_rows, force_srate)
   1211 print(bod)
   1212 print(num_data_pts)
-> 1213 output["data"].append(np.memmap(
   1214     self.datafile,
   1215     dtype="int16",
   1216     mode="r",
   1217     offset=bod,
   1218     shape=(num_data_pts, self.basic_header["ChannelCount"]),
   1219     order="C"
   1220 ))
   1221 # memmap moves the file pointer inconsistently depending on platform and numpy version
   1222 curr_loc = self.datafile.tell()

File ~/anaconda3/envs/env_nelson/lib/python3.13/site-packages/numpy/core/memmap.py:268, in memmap.__new__(subtype, filename, dtype, mode, offset, shape, order)
    266 bytes -= start
    267 array_offset = offset - start
--> 268 mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    270 self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,
    271                        offset=array_offset, order=order)
    272 self._mmap = mm

ValueError: mmap length is greater than file size

debugging it, I see that the file being red is giving for the first data package:

• bod -> 2435 (which seems to be correct)
• num_data_pts -> 0 (which seems to be wrong)

inspecting the binary data, I see that indeed the 4 bytes encoding the number of samples is giving 0!

Is this a valid value for the system, or was the file somehow crrupted?

[luiztauffer]

@dkluger @cboulay could you help me figure this out?
Apparently the users in the lab are able to see the raw voltage traces in the nPlay software, but these files do not seem to match the Blackrock file specs (the zero-data points problem), and therefore fail to be read by the Python Utilities.
I could share the data files with you, if that would help!