diff --git a/docs/cli.md b/docs/cli.md index 0e4a453..e3f1e33 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -8,6 +8,6 @@ The `lslautobids` command-line interface provides the main entry point for the a #### Key Points -1. `lslautobids en-proj-config` and `lslautobids gen-dv-config` commands generate configuration files for the project and Dataverse, respectively. This allows users to set up their project and Dataverse connection details easily before running the conversion and upload process +1. `lslautobids gen-proj-config` and `lslautobids gen-dv-config` commands generate configuration files for the project and Dataverse, respectively. This allows users to set up their project and Dataverse connection details easily before running the conversion and upload process 2. The `lslautobids run` command executes the main conversion and upload process, using the configurations generated earlier. This command runs the entire pipeline from reading XDF files, converting them to BIDS format, integrating with DataLad, and uploading to Dataverse. 3. The `lslautobids help` command provides usage information for the CLI, listing available commands and their descriptions. diff --git a/docs/dataverse_integration.md b/docs/dataverse_integration.md index c35cf95..91d143f 100644 --- a/docs/dataverse_integration.md +++ b/docs/dataverse_integration.md @@ -1,6 +1,14 @@ ### Dataverse Integration -#### 1. Dataverse Dataset Creation (`dataverse_dataset_create.py`) +#### 1. Generating dataset JSON Metadata (`generate_dataset_json.py`) + +This module generates the `dataset.json` file required for creating a dataset in Dataverse. The function performs the following steps: + +1. Gathers metadata from the project configuration file (_config.toml) such as title, authors, description, license, etc. +2. Constructs a JSON structure that conforms to the Dataverse dataset metadata schema. This includes fields like title, author list, description, keywords, license, etc. +3. Writes the constructed JSON to a file named `dataset.json` in the project directory. This file is then used when creating the dataset in Dataverse. + +#### 2. Dataverse Dataset Creation (`dataverse_dataset_create.py`) This module handles the creation of a new dataset in Dataverse using the `pyDataverse` library. The function performs the following steps: 1. Initialize `dataset.json` file path and read the JSON content. (See section) @@ -12,7 +20,7 @@ This module handles the creation of a new dataset in Dataverse using the `pyData 5. If no existing dataset is found, we create a new dataset using `api.create_dataset(parent_dataverse_name, ds.json())`. We then populate the returned dataset ID and DOI in the project configuration file (_config.toml) for using in future runs. 6. This function returns the dataset DOI and status code ( 1= dataverse dataset exists, 0= new dataset created) -#### 2. Linking DataLad to Dataverse (`link_datalad_dataverse.py`) +#### 3. Linking DataLad to Dataverse (`link_datalad_dataverse.py`) This module links the local DataLad dataset to the remote Dataverse dataset as a sibling. The function performs the following steps: 1. It first checks if the Dataverse is already created in the previous runs or it is just created in the current run (flag==0). If flag==0, it proceeds to link the DataLad dataset to Dataverse. 2. It runs the command `datalad add-sibling-dataverse dataverse_base_url doi_id`. This command adds the Dataverse as a sibling to the local DataLad dataset, allowing for synchronization and data management between the two. For lslautobids, we currently only allow to deposit data to Dataverse. In future version, we shall also add user controlled options for adding other siblings like github, gitlab, OpenNeuro, AWS etc. @@ -21,16 +29,7 @@ We chose Dataverse as it serves as both a repository and a data sharing platform Dataverse also provides features like versioning, but only after we publish the dataset. In our case, we keep the dataset in draft mode until we are ready to publish it (i.e. until all the participants/subjects data is uploaded). So we use DataLad for version control during the development and conversion phase to assure complete provenance of the dataset. - -#### 4 Generating dataset JSON Metadata (`generate_dataset_json.py`) - -This module generates the `dataset.json` file required for creating a dataset in Dataverse. The function performs the following steps: - -1. Gathers metadata from the project configuration file (_config.toml) such as title, authors, description, license, etc. -2. Constructs a JSON structure that conforms to the Dataverse dataset metadata schema. This includes fields like title, author list, description, keywords, license, etc. -3. Writes the constructed JSON to a file named `dataset.json` in the project directory. This file is then used when creating the dataset in Dataverse. - -#### 5. Upload to Dataverse (`upload_to_dataverse.py`) +#### 4. Upload to Dataverse (`upload_to_dataverse.py`) This module handles the uploading of files from the local DataLad dataset to the remote Dataverse dataset. The function performs the following steps: 1. It runs the command `datalad push --to dataverse` to push the files from the local DataLad dataset to the linked Dataverse dataset. This command uploads all changes (new files, modified files) to Dataverse. diff --git a/docs/testing.md b/docs/testing.md index d6673b7..d4d1cc6 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -14,6 +14,6 @@ Tests will be added continuously as new features are added and existing features ### Running Tests -To run the tests, we recommend to use `uv run pytest` (caveat, for some reason sometimes tests fail if they are all run at the same time. you can then run them via `uv run pytest tests/testcase/test_main_functionality` and they will work). +Before running the tests with `uv`, install the project in editable mode using `uv pip install --editable ./`. Then run the tests with `uv run pytest` (caveat, sometimes running all tests at once fails; in that case, you can run only the main functionality tests via `uv run pytest tests/test_main_functionality`). These tests ensure that each component functions as expected and that the overall pipeline works seamlessly. This tests will also be triggered automatically on each push or PR to the main repository using GitHub Actions. \ No newline at end of file diff --git a/sample_data/bids/test-project/.bidsignore b/sample_data/bids/test-project/.bidsignore new file mode 120000 index 0000000..bab93fc --- /dev/null +++ b/sample_data/bids/test-project/.bidsignore @@ -0,0 +1 @@ +../../../.git/annex/objects/Wv/vp/SHA256E-s27--c74b0467cfa3fdcdd8a215bdb896d8f4830833eb6dc0be8b330e19bef34c2da8/SHA256E-s27--c74b0467cfa3fdcdd8a215bdb896d8f4830833eb6dc0be8b330e19bef34c2da8 \ No newline at end of file diff --git a/sample_data/bids/test-project/README b/sample_data/bids/test-project/README new file mode 120000 index 0000000..1650d91 --- /dev/null +++ b/sample_data/bids/test-project/README @@ -0,0 +1 @@ +../../../.git/annex/objects/Wg/z1/SHA256E-s692--e0b5231ca06b97a1597194ee398df2347842e64f45398e62bf420c6ba8694b63/SHA256E-s692--e0b5231ca06b97a1597194ee398df2347842e64f45398e62bf420c6ba8694b63 \ No newline at end of file diff --git a/sample_data/bids/test-project/dataset_description.json b/sample_data/bids/test-project/dataset_description.json new file mode 120000 index 0000000..fa51fb6 --- /dev/null +++ b/sample_data/bids/test-project/dataset_description.json @@ -0,0 +1 @@ +../../../.git/annex/objects/mG/qX/SHA256E-s156--0f06f9bade6916560df16da8fd82ac4e9f47006e5dd8e414fcbdfdc8bdc01469.json/SHA256E-s156--0f06f9bade6916560df16da8fd82ac4e9f47006e5dd8e414fcbdfdc8bdc01469.json \ No newline at end of file diff --git a/sample_data/bids/test-project/participants.json b/sample_data/bids/test-project/participants.json new file mode 120000 index 0000000..8852c57 --- /dev/null +++ b/sample_data/bids/test-project/participants.json @@ -0,0 +1 @@ +../../../.git/annex/objects/1V/v0/SHA256E-s756--4b8aeb7120262dc2ff8cb93d218fdcc09062a65f0ec300afa2e67bb6c81d95dc.json/SHA256E-s756--4b8aeb7120262dc2ff8cb93d218fdcc09062a65f0ec300afa2e67bb6c81d95dc.json \ No newline at end of file diff --git a/sample_data/bids/test-project/participants.tsv b/sample_data/bids/test-project/participants.tsv new file mode 120000 index 0000000..8cc5d87 --- /dev/null +++ b/sample_data/bids/test-project/participants.tsv @@ -0,0 +1 @@ +../../../.git/annex/objects/1f/w4/SHA256E-s73--049a14247f46da037432e97b3c4bcce9e8b67f9b1d5efdd2d9ac3d544666e86d.tsv/SHA256E-s73--049a14247f46da037432e97b3c4bcce9e8b67f9b1d5efdd2d9ac3d544666e86d.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sourcedata/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-002_eeg_raw.xdf b/sample_data/bids/test-project/sourcedata/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-002_eeg_raw.xdf new file mode 120000 index 0000000..c4864c8 --- /dev/null +++ b/sample_data/bids/test-project/sourcedata/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-002_eeg_raw.xdf @@ -0,0 +1 @@ +../../../../../../../.git/annex/objects/z9/ZG/SHA256E-s2105239--5539ce3b5199e21c9c5ad5d9b9092ffdd261a5c0f2f4d19cca01a85c1ecbd5cc.xdf/SHA256E-s2105239--5539ce3b5199e21c9c5ad5d9b9092ffdd261a5c0f2f4d19cca01a85c1ecbd5cc.xdf \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_channels.tsv b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_channels.tsv new file mode 120000 index 0000000..120ee6d --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_channels.tsv @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/X2/gF/SHA256E-s7522--e456869358aaff22c0b5c69ba1aa0b9c82e333eac4539dddfe7d4bf7b7d1fd05.tsv/SHA256E-s7522--e456869358aaff22c0b5c69ba1aa0b9c82e333eac4539dddfe7d4bf7b7d1fd05.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.json b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.json new file mode 120000 index 0000000..a244ee6 --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.json @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/7p/WW/SHA256E-s502--c9d08adf342446b71c75ddd9c39abad81064e5286a97103902ef94ae74c0136f.json/SHA256E-s502--c9d08adf342446b71c75ddd9c39abad81064e5286a97103902ef94ae74c0136f.json \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.set b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.set new file mode 120000 index 0000000..74a214b --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_eeg.set @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/Fg/8j/SHA256E-s2168304--493e700883cb1393e36fcf5eee2896857be75e0aeff57fc48dc8e28a039b3697.set/SHA256E-s2168304--493e700883cb1393e36fcf5eee2896857be75e0aeff57fc48dc8e28a039b3697.set \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.json b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.json new file mode 120000 index 0000000..7469993 --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.json @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/gf/wx/SHA256E-s767--daf047ec40acc225a37db09baa7e3ec9e01d7011d92784f44394c3f77c3640b3.json/SHA256E-s767--daf047ec40acc225a37db09baa7e3ec9e01d7011d92784f44394c3f77c3640b3.json \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.tsv b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.tsv new file mode 120000 index 0000000..85c88b8 --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/eeg/sub-999_ses-001_task-freeviewing_run-2_events.tsv @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/8K/8Q/SHA256E-s164--4fb00c45c04814077930cc9d4952b01695d4a675302ba7e4985141c3fdc3691d.tsv/SHA256E-s164--4fb00c45c04814077930cc9d4952b01695d4a675302ba7e4985141c3fdc3691d.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/misc/experiment.tar.gz b/sample_data/bids/test-project/sub-999/ses-001/misc/experiment.tar.gz new file mode 120000 index 0000000..938569b --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/misc/experiment.tar.gz @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/JZ/JP/SHA256E-s570--f28e147c8848e869f9016636d97ca06d377b39d733f0bfe071f1d97da2dc0a9d.tar.gz/SHA256E-s570--f28e147c8848e869f9016636d97ca06d377b39d733f0bfe071f1d97da2dc0a9d.tar.gz \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_labnotebook.tsv b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_labnotebook.tsv new file mode 120000 index 0000000..f75d150 --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_labnotebook.tsv @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/JG/4m/SHA256E-s70--ec8504be25531c7c596a5be160adb2b7fa85e8fb018a08eccb8e647431b54c36.tsv/SHA256E-s70--ec8504be25531c7c596a5be160adb2b7fa85e8fb018a08eccb8e647431b54c36.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_participantform.tsv b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_participantform.tsv new file mode 120000 index 0000000..b53b7ec --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_participantform.tsv @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/qq/Kf/SHA256E-s80--83c1efe47b82e4a4fcc7f7757db47f9c7871c25fa26b47d6d3b355e9b5c28c86.tsv/SHA256E-s80--83c1efe47b82e4a4fcc7f7757db47f9c7871c25fa26b47d6d3b355e9b5c28c86.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_beh.tsv b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_beh.tsv new file mode 120000 index 0000000..f8eb5bc --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_beh.tsv @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/Qf/Pq/SHA256E-s11222--52cd7aee1f566759bea2d11e44a3580a28b2a25de97338e8a25ba71d8439ae43.tsv/SHA256E-s11222--52cd7aee1f566759bea2d11e44a3580a28b2a25de97338e8a25ba71d8439ae43.tsv \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_physio.edf b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_physio.edf new file mode 120000 index 0000000..031fdc6 --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/misc/sub-999_ses-001_task-Default_physio.edf @@ -0,0 +1 @@ +../../../../../../.git/annex/objects/8v/jP/SHA256E-s652093--6336057e3a53ea4bff5efc5d5222f290ffad1ace2e0165a2eba1b0f778a7a336.edf/SHA256E-s652093--6336057e3a53ea4bff5efc5d5222f290ffad1ace2e0165a2eba1b0f778a7a336.edf \ No newline at end of file diff --git a/sample_data/bids/test-project/sub-999/ses-001/sub-999_ses-001_scans.tsv b/sample_data/bids/test-project/sub-999/ses-001/sub-999_ses-001_scans.tsv new file mode 120000 index 0000000..597216b --- /dev/null +++ b/sample_data/bids/test-project/sub-999/ses-001/sub-999_ses-001_scans.tsv @@ -0,0 +1 @@ +../../../../../.git/annex/objects/W5/f6/SHA256E-s100--a9cc369f2308442e553da03cfdeee20150fe1b90d4b80109663448ee0d1d0df7.tsv/SHA256E-s100--a9cc369f2308442e553da03cfdeee20150fe1b90d4b80109663448ee0d1d0df7.tsv \ No newline at end of file diff --git a/tests/pytest-autobids_config.yaml b/tests/pytest-autobids_config.yaml index bd9dd5b..c18e2aa 100644 --- a/tests/pytest-autobids_config.yaml +++ b/tests/pytest-autobids_config.yaml @@ -1,3 +1,3 @@ -BIDS_ROOT: /home/behinger/projects/LSLAutoBIDS/tests/data/bids -PROJECT_OTHER_ROOT: /home/behinger/projects/LSLAutoBIDS/tests/data/project_other -PROJECT_ROOT: /home/behinger/projects/LSLAutoBIDS/tests/data/projects +BIDS_ROOT: /Users/xuyg/GitHub/LSLAutoBIDS/tests/data/bids +PROJECT_OTHER_ROOT: /Users/xuyg/GitHub/LSLAutoBIDS/tests/data/project_other +PROJECT_ROOT: /Users/xuyg/GitHub/LSLAutoBIDS/tests/data/projects diff --git a/tests/test_main_functionality/test_main_functionality.py b/tests/test_main_functionality/test_main_functionality.py index 719d82c..095ef0a 100644 --- a/tests/test_main_functionality/test_main_functionality.py +++ b/tests/test_main_functionality/test_main_functionality.py @@ -31,6 +31,9 @@ def test_process_main_functionality(setup_project, monkeypatch): monkeypatch_paths(monkeypatch,paths) + from lslautobids.config_logger import get_logger + logger = get_logger(project_name) + logger.debug(f" Starting test_process_main_functionality in {project_name} ") # Reset sys.argv to something that lslautobids.main.main() expects # this effectively removes the -c from setup_project diff --git a/tests/test_old_suffix/test_old_suffix.py b/tests/test_old_suffix/test_old_suffix.py index d1a44bd..8b35366 100644 --- a/tests/test_old_suffix/test_old_suffix.py +++ b/tests/test_old_suffix/test_old_suffix.py @@ -28,6 +28,10 @@ def test_process_new_files_with_old_suffix(setup_project, monkeypatch): paths = get_root_paths(__file__) monkeypatch_paths(monkeypatch,paths) + + from lslautobids.config_logger import get_logger + logger = get_logger(project_name) + logger.debug(f" Starting test_process_new_files_with_old_suffix in {project_name} ") # Reset sys.argv to something that lslautobids.main.main() expects sys.argv = [