help for working with Febus files

[SoumayaLatour]

Hello,
Thank you very much for the package and the starter guide.

I have a problem for reading Febus files, that I think is related to the question of @Linvill a few monthes ago:
My file contains data at 400Hz that are 1hour long with 417 channels. So it is composed of 3600 blocks of 420x417 values (overlap being 10 sample on each side).
I tried to open one file by typing :
da = xdas.open_dataarray("my_file.h5", engine="febus",overlaps=(10,10),offset=10)
but it never finishes.
So I tried to run the "febus" read routine line by line with my file, to see where it freezes. The problem appears at the last line:

return(concatenate(dc,"time")

So I ran xdas.concatenate(dc,"time",verbose=True), and I see that for the first seconds the it/s is around 60 or so, but then it decreases up to 1it/s.

when I look at dc, it looks like this:

<xdas.DataArray (time: 400, distance: 417)>
 VirtualSource: 651.6KB (float32)
 Coordinates:
   * time (time): 2025-02-14T06:17:20.999 to 2025-02-14T06:17:21.997
   * distance (distance): 0.000 to 1996.800,
 <xdas.DataArray (time: 400, distance: 417)>
 VirtualSource: 651.6KB (float32)
 Coordinates:
   * time (time): 2025-02-14T06:17:21.999 to 2025-02-14T06:17:22.997
   * distance (distance): 0.000 to 1996.800,
 <xdas.DataArray (time: 400, distance: 417)>
 VirtualSource: 651.6KB (float32)

So I have tried the fix you gave to @Linvill :
typed pip install git+https://github.com/xdas-dev/xdas.git@dev in the conda environnemnt where I had installed xdas, closed spyder, reopened it in this environment, and tryied again to read the file, but the problem was still there.

However, as I am a beginner with python and conda environnement, I am not completely sure I am running the fixed version of xdas or not, or what I could do next to solve the problem.

Thank you for any help or tips!

PS:I can share one of my data files if needed.

[atrabattoni]

Hi @SoumayaLatour,

Thank's for your question. Before we continue this disscussion could you try installing the dev version like this:

pip install "git+https://github.com/xdas-dev/xdas.git@dev" --force-reinstall

The thing is that maybe I should increase the version number of the dev branch at each release because otherwise with a simple pip install pip consider that the dev version is the same than the actual one and does nothing.

Let me know if this solves your issue !

BTW, are you sure about offset=10? With the latest version of the Febus file format i think the timestamp is in the middle so offset=210 should be correct. This is the value automatically recovered by Xdas if no offset nor overlaps are provided.

[SoumayaLatour]

Thank you for your fast answer!
I forced the re-installation this morning as you instructed, and was able to open the file! It took approximately 8s to open, not sure if this is the expected duration for such a file.
However, once the file is open, python is really slow, as if the oppening occupy too much resources BUT this is probably due to my computer being almost full, I will check on Friday after liberating some space and let you know.
Regarding the offset and overlaps value, thank you for raising the issue to my attention, I do not know what are the good values for my case, is there a way to check for sure?
Thanks again!

[atrabattoni]

Well 8s is definitely not super fast, but the Febus engine of Xdas is known to be a bit slow because of the overlap management (there is room for improvement). Yet I thing it's workable like this. I encourage you, once you've virtually oppened and consolidated your dataset, to save it as a master file with the dc.to_netcdf("master_file_path.nc", virtual=True) so that you avoid the computation cost of this operation.

[atrabattoni]

General comment for all users:

• overlaps: seems to be correctly recovered for all Febus files version. No need to specify it.
• offset: for older version, setting offset=0 seems to be the correct way. For modern version, Xdas default behaviour seem to work well.

There might be some intermediate version with weird behaviour be carefull 😄 .

[atrabattoni]

I made a new release (0.2.3) including all the Febus fixes. The following should work now:

pip install -U xdas

[SoumayaLatour]

The file is 2.5Go. It looks like once the file is opened, python uses 1.5Go of memory.
Here is one of my file [https://filesender.renater.fr/?s=download&token=d3887bbf-a834-422a-9c64-19b7c69c1ba4]
By the way, not sure if it is relevant, but I use python via the command line of spyder.
I have tried without defiing the overlap, it does not changes this behavious.

[atrabattoni]

Okay I could open the file without any issue in a reasonable amount of time and everything keeps being reactive. Could you try to run the following script:

import platform
import sys
import warnings
from time import perf_counter

import psutil
import xdas as xd

warnings.filterwarnings("ignore")

path = "data/SR.hdf5"  # change the path here

start_time = perf_counter()
da = xd.open_dataarray(path, engine="febus")
print(f"Elapsed time: {perf_counter() - start_time:.2f} seconds")

process = psutil.Process()
ram_used_mb = process.memory_info().rss / (1024**2)
print(f"RAM used by Python: {ram_used_mb:.2f} MB")

print(
    f"Python : {sys.version.split("|")[0].strip()} | "
    f"OS: {platform.system()} | "
    f"Architecture {platform.machine()}"
)

On my machine I get:

Elapsed time: 2.83 seconds
RAM used by Python: 301.23 MB
Python : 3.13.5 | OS: Linux | Architecture x86_64

[SoumayaLatour]

Thank you for the help,
when I run the script I get:
Elapsed time: 3.44 seconds RAM used by Python: 343.87 MB Python : 3.13.5 | OS: Darwin | Architecture x86_64

But I have made some tests, the problem appears to originates from the Spyder interactive environnement:
When I run this script in Spyder, it prints the results shown above, but then lags for approx 10 s before giving access to the prompt again in the console. And then everything else in spyder lags, as long as I have the variable da in memory. And the MacOS activity monitor indicates that Python3.13 occupy 1.5Go of "memory" (not sure if it is RAM or not).
In contrast, when I open python in a terminal, and execute the script line by line, then da is in memory, but afterward nothing lags, and the activity monitor indicates that Python3.13 uses 190Mo.

So I think I will just construct my data collection without using Spyder, save it and then see if I can use it from spyder without this problem.

[atrabattoni]

Okay interesting I will open an issue about this. Might be that a spyder component such as the variable explorer interfere within the process and slow down things. Thanks you for repporting that.

[SoumayaLatour]

Yes I confirm, the problem comes from the variable explorer of Spyder. It seems to be related to this old issue: link.

For anyone encountering the same problem, I solved it by modifying the options of the variable explorer in Spyder: in the drop down menu of the variable explorer, I checked the "Exclude unsupported data types" option. Now it does not lag anymore. Python still uses 1.5Go of memory, but I guess this is what is needed for Spyder to run.

Thank you for your help @atrabattoni !