toolviper.utils.data.list_files print truncated name that there is no way to access or guess

[kgolap]

See above image
There is no way for me to download the file as it claims it does not exist
list_files()
download(file="3c286_Band6_5chans_lsrk_robust_0.5_niter_99.fits")

and download does not recognize wildcard * and the help does not describe any kwargs that can be used

[jrhosk]

Thanks Kumar, I'll take a look at this in the next few days.

[jrhosk]

@kgolap I'm trying to look at this today since I have some free time. I'll look at two things: adding a wildcard option and not truncating the filenames if possible. In the meantime, there is a function already available you will find useful. Instead of list_files() you can use get_files() to get a python list of the available file names. See the module documentation here : https://github.com/casangi/toolviper/blob/main/docs/download_example.ipynb

[jrhosk]

@kgolap I made a ton of changes and essentially changed how it woks completely under the hood. The list_files() function can now print an ascii table, let the user choose the truncation level, and if you are using it in a notebook, you will get an interactive table with a search function among other features. In addition, you can grab the output object of the function, which is a data frame object and has all the functionality that comes with that, including .filter(...), .group_by(...) and .select(...) functions. You can take a look at the full documentation here. I have also updated the notebook that I linked above to include simple examples. Please pull the branch and test is yourself and give feedback. When you are happy, I will merge it and do a new release.

[kgolap]

@jrhosk . Thanks !
The dataFrame output of list_files() is very useful and sufficient for my use, solves the bug/problem I had.

I don't see any parameter or help in list_files() to choose truncation level you .

[jrhosk]

@kgolap ahhh did you the link above was to main I think. The modified function and example is on the branch here . It only works with the ascii table output, but I think that is the only mode it is needed on. You can see it at the bottom of the notebook where I showed a with/without truncation example. The ascii table is mess though in notebooks, I think because the horizontal settings seem to be constricted; looks good in terminal though where it is intended.

I'll go ahead and merge this and do a release soon.

[kgolap]

I did pull that branch and "pip install -e ." The list_files function does return a polars dataframe, which solves my problem, (as compared to None before) but I see parameter truncate...but no help on possible values it can take or what the default means.

list_files(truncate=None) -> polars.dataframe.frame.DataFrame

[Jan-Willem]

@jrhosk — Is my understanding correct that list_files returns a Polars DataFrame? To avoid “dependency hell,” we should minimize adding new dependencies where possible. We already depend on pandas via xarray, and some xradio functions already return pandas DataFrames (for example, in the processing_set summary accessor). From a usability perspective, it would be beneficial to standardize on a single DataFrame API.

Would it be possible to replace the Polars dependency with pandas in this case?

[jrhosk]

Yes, I can replace it. I would replace all the pandas with polars honestly, its faster and has essentially the same API. You make a good point though, i will reluctantly change it.