Where can I download the dataset?

[datong-new]

Hello, thanks for the wonderful work!

Can you give more details about the dataset? And where can I download the dataset?

Thank you!

[maxjcohen]

Hi,

You can find a sample of the dataset, as well as a brief description, as an open data challenge, in csv format. You will have to transpose it to npz format, or use a custom pytorch dataset (see the challenge demo repo), in order to use the notebooks.

Bests

[HuskyLens]

Hi,
Would you like to share the npz file? As the data structure from Open Data Challenge seems different from yours.
See the difference:
Yours
Origin

[maxjcohen]

Hi, I can't share a npz file containing any other data than the ones uploaded on the data challenge, as it would go against the very rules of the challenge.
The structure of the labels is different, but that shouldn't be an issue if you just want to convert the csv dataset to npz, as the code was written with these possible modifications in mind. Just load the csv with the OzeDataset class, and export R, Z and X using np.savez. You're aiming at this kind of data structure.

[francisduan]

Hi do you have any code that could transform the csv to npz, I am not sure what we should include in the npz

[maxjcohen]

Once again, all needed information are present in the challenge benchmark repo, but to prevent further questions on the dataset I have drafted a function to convert csv to npz.

[DanielAtKrypton]

Dear @maxjcohen , I joined the challenge 28, downloaded the following files:

• x_train_LsAZgHU.csv
• y_train_EFo1WyE.csv
• x_test_QK7dVsy.csv

Then I copied csv2npz script to utils folder within the project.
Then I created and ran the following python script at project's root folder:

from src.utils.csv2npz import csv2npz

csv2npz('datasets/x_train_LsAZgHU.csv', 'datasets/y_train_EFo1WyE.csv')

But unfortunately it errored as can be seen below.

Traceback (most recent call last):
  File "/home/<username>/Workspaces/Python/transformer/generateNpz.py", line 3, in <module>
    csv2npz('datasets/x_train_LsAZgHU.csv', 'datasets/y_train_EFo1WyE.csv')
  File "/home/<username>/Workspaces/Python/transformer/src/utils/csv2npz.py", line 21, in csv2npz
    R = x[labels["R"]].values
  File "/home/<username>/.virtualenvs/.env/lib/python3.7/site-packages/pandas/core/frame.py", line 2806, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "/home/<username>/.virtualenvs/.env/lib/python3.7/site-packages/pandas/core/indexing.py", line 1553, in _get_listlike_indexer
    keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing
  File "/home/<username>/.virtualenvs/.env/lib/python3.7/site-packages/pandas/core/indexing.py", line 1646, in _validate_read_indexer
    raise KeyError(f"{not_found} not in index")
KeyError: "['initial_temperature', 'roof_1_thickness_3'] not in index"

[maxjcohen]

Hi, this error means that the index "initial_temperature" and "roof_thickness_3" are not present in the challenge dataset. Indeed, if you take the original labels.json, these values are not present, because they were not intended to be used in the challenge.

In order to solve your error, I recommend using the original labels file from the benchmark repo.

[DanielAtKrypton]

I created a pull request #6 with some improvements I came up with up to now, it might be useful to merge @maxjcohen, please advise.