Multisession modeling functionality for AutoLFADS by claytonwashington · Pull Request #152 · cunningham-lab/neurocaas

claytonwashington · 2023-09-01T15:19:34Z

No description provided.

…re update at 2023-10-03 16:52:48.997520

…r update at 2023-10-03 16:52:49.268505. Purpose: test. AMI: ami-05ddab33dd696e7d0

…r update at 2023-10-04 11:00:56.540733. Purpose: setup before first single run. AMI: ami-076b9ecc29502b344

…r update at 2023-10-09 17:07:12.925453. Purpose: upgraded CUDA. AMI: ami-0295578a40774d483

…r update at 2023-10-09 17:40:31.179587. Purpose: updated cuda and got sklearn. AMI: ami-0163b17cdfa51a9f5

…r update at 2023-10-10 17:08:18.822096. Purpose: final test on local single and multi. AMI: ami-0ccfff108ae8de464

cellistigs · 2023-10-27T17:30:43Z

Confirmed unit tests passed on local machine. Moving on to tests with AWS SAM.

cellistigs · 2023-10-27T17:34:28Z

Sam package built/Template validated.

$ cd ~/neurocaas/ncap_iac/ncap_blueprints/iac_utils/
$ bash fullbuild.sh ../autolfads-torch
$ sam validate --template-file

cellistigs · 2023-10-27T20:48:28Z

Built AWS sam package successfully (docker failing before- updated fullbuild.sh to use appropriate context.)

cellistigs · 2023-10-27T20:58:26Z

Need multisession parameter for this analysis to pass in config file.

cellistigs · 2023-10-27T21:29:59Z

Tested standard implementation. Across various errors (different environment variables, misspecified arns) preserved behavior of shutting down instances as expected.

cellistigs · 2023-10-27T21:42:38Z

At the moment, ncap_iac/protocols/submit_start.py has both process_upload_dev and process_upload_multisession. It appears these are identical save for the submission class passed to them. It would be good to refactor this so we can pass a submission object- since these functions take care of error handling having a single function wherever possible will simplify testing in the future.

cellistigs · 2023-10-27T22:28:50Z

The following functionality has been tested for reliability of cloud management systems, validated through sam local invoke calls. Note that we only check MainLambda here, as the behavior of the postprocessing lambda file does not have to change.

failing to pass a multisession parameter
passing multisession:False with a single dataset in the inputs folder.
passing multisession:True with a single dataset in the inputs folder.
passing multisession:True with datasets in multisession_test (directory passed to submit file) (launches one instance)
passing multisession:False with datasets in multisession_test (directory passed to submit file) (launches one instance)
passing multisession:True with datasets in multisession_test (individual files passed to submit file) (launches two instances)
passing multisession:False with datasets in multisession_test (individual files passed to submit file) (launches two instance)

cellistigs · 2023-10-27T22:29:00Z

#deploy:autolfads-torch

cellistigs · 2023-10-27T22:31:22Z

We have some issues with logging- at least on AWS-SAM, we only ever see messages from logging start of the first instance. Check this in deployment. Note we have only checked the above functions for cloud reliability, so the above formats should be tested for expected functionality in all combinations of inputs listed above.

We should also refactor: At the moment, ncap_iac/protocols/submit_start.py has both process_upload_dev and process_upload_multisession. It appears these are identical save for the submission class passed to them. It would be good to refactor this so we can pass a submission object- since these functions take care of error handling having a single function wherever possible will simplify testing in the future.

cellistigs · 2023-10-27T22:48:37Z

#deploy:autolfads-torch

cellistigs · 2023-10-27T22:52:18Z

#deploy:autolfads-torch

…r update at 2023-10-31 14:18:09.494446. Purpose: multisession entry no longer needed in config. AMI: ami-0699e307bc81482db

cellistigs · 2023-10-31T18:34:23Z

#deploy:autolfads-torch

…r update at 2023-11-10 15:49:37.826420. Purpose: fixed lfads-torch import. AMI: ami-0893ac69ba200a546

cellistigs · 2023-11-13T19:26:38Z

#deploy:autolfads-torch

…r update at 2023-11-16 13:55:30.184299. Purpose: fixed issues with access for root/SSM user. AMI: ami-017465330c3757649

…r update at 2023-11-16 14:25:10.865461. Purpose: root path fix fix?. AMI: ami-0754772739da3a57d

cellistigs · 2023-11-16T20:32:35Z

#deploy:autolfads-torch

…r update at 2023-11-28 17:12:13.330582. Purpose: fixed single dataset run; command broken for multisession. AMI: ami-024a73c9849ee99e4

…r update at 2023-12-01 21:09:51.895815. Purpose: new multisession post-tutorial. AMI: ami-0825772e0ac2e67c8

…r update at 2023-12-05 22:16:10.678782. Purpose: successful ssm test on multisession via nc_contrib hotfix 51. AMI: ami-0cebcfea203997fcf

cellistigs · 2023-12-06T22:52:33Z

#deploy:autolfads-torch

…ed on web

cellistigs

Most of the changes here look good- however I have some suggestions regarding code structure and logging.

ncap_iac/protocols/submit_start.py

…b interface

cellistigs · 2023-12-12T21:28:41Z

#deploy:autolfads-torch

cellistigs · 2023-12-15T19:56:49Z

#deploy:autolfads-torch

cellistigs · 2023-12-15T20:27:12Z

#deploy:autolfads-torch

cellistigs · 2023-12-15T20:41:59Z

#deploy:autolfads-torch

cellistigs · 2023-12-15T20:58:45Z

#deploy:autolfads-torch

cellistigs · 2023-12-15T21:02:54Z

#deploy:autolfads-torch

claytonwashington added 11 commits August 25, 2023 11:29

attempt 1 at full changes for msm

fb2e64b

stash for testing

fe77fab

post-testing changes

2e50d16

test

f92f4ff

fix

7415356

automatic commit to document pipeline stack_config_template.json befo…

f185ba7

…re update at 2023-10-03 16:52:48.997520

automatic commit to document pipeline stack_config_template.json afte…

0cb782e

…r update at 2023-10-03 16:52:49.268505. Purpose: test. AMI: ami-05ddab33dd696e7d0

automatic commit to document pipeline stack_config_template.json afte…

4ea3dec

…r update at 2023-10-04 11:00:56.540733. Purpose: setup before first single run. AMI: ami-076b9ecc29502b344

automatic commit to document pipeline stack_config_template.json afte…

7700632

…r update at 2023-10-09 17:07:12.925453. Purpose: upgraded CUDA. AMI: ami-0295578a40774d483

automatic commit to document pipeline stack_config_template.json afte…

6f3c55d

…r update at 2023-10-09 17:40:31.179587. Purpose: updated cuda and got sklearn. AMI: ami-0163b17cdfa51a9f5

automatic commit to document pipeline stack_config_template.json afte…

33edce2

…r update at 2023-10-10 17:08:18.822096. Purpose: final test on local single and multi. AMI: ami-0ccfff108ae8de464

claytonwashington added 2 commits October 31, 2023 11:11

no multisession field in config defaults to false

b031c66

automatic commit to document pipeline stack_config_template.json afte…

025c8f8

…r update at 2023-10-31 14:18:09.494446. Purpose: multisession entry no longer needed in config. AMI: ami-0699e307bc81482db

automatic commit to document pipeline stack_config_template.json afte…

39badb9

…r update at 2023-11-10 15:49:37.826420. Purpose: fixed lfads-torch import. AMI: ami-0893ac69ba200a546

claytonwashington added 2 commits November 16, 2023 13:55

automatic commit to document pipeline stack_config_template.json afte…

6400428

…r update at 2023-11-16 13:55:30.184299. Purpose: fixed issues with access for root/SSM user. AMI: ami-017465330c3757649

automatic commit to document pipeline stack_config_template.json afte…

1540f23

…r update at 2023-11-16 14:25:10.865461. Purpose: root path fix fix?. AMI: ami-0754772739da3a57d

claytonwashington added 4 commits November 28, 2023 15:22

drive run_main.py with run_main.sh

3207d97

automatic commit to document pipeline stack_config_template.json afte…

51303b6

…r update at 2023-11-28 17:12:13.330582. Purpose: fixed single dataset run; command broken for multisession. AMI: ami-024a73c9849ee99e4

automatic commit to document pipeline stack_config_template.json afte…

a02e41d

…r update at 2023-12-01 21:09:51.895815. Purpose: new multisession post-tutorial. AMI: ami-0825772e0ac2e67c8

automatic commit to document pipeline stack_config_template.json afte…

1b2d533

…r update at 2023-12-05 22:16:10.678782. Purpose: successful ssm test on multisession via nc_contrib hotfix 51. AMI: ami-0cebcfea203997fcf

fix self.data_name_list in submission object for multisession submitt…

66762a6

…ed on web

cellistigs reviewed Dec 12, 2023

View reviewed changes

ncap_iac/protocols/submit_start.py Outdated Show resolved Hide resolved

ncap_iac/protocols/submit_start.py Outdated Show resolved Hide resolved

ncap_iac/protocols/submit_start.py Outdated Show resolved Hide resolved

ncap_iac/protocols/submit_start.py Show resolved Hide resolved

incorporated Taiga's suggestions to code enabling multisession via we…

4d3fb92

…b interface

debug logs for multisession logic

e40d9fb

debug messages in handler

dd0885d

moved debug print statement in multi handler

beac528

changed True to boolean

8808003

changed True to boolean again

f6842d8

Conversation

claytonwashington commented Sep 1, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cellistigs commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 27, 2023

Uh oh!

cellistigs commented Oct 31, 2023

Uh oh!

cellistigs commented Nov 13, 2023

Uh oh!

cellistigs commented Nov 16, 2023

Uh oh!

cellistigs commented Dec 6, 2023

Uh oh!

cellistigs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cellistigs commented Dec 12, 2023

Uh oh!

cellistigs commented Dec 15, 2023

Uh oh!

cellistigs commented Dec 15, 2023

Uh oh!

cellistigs commented Dec 15, 2023

Uh oh!

cellistigs commented Dec 15, 2023

Uh oh!

cellistigs commented Dec 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cellistigs commented Oct 27, 2023 •

edited

Loading

cellistigs commented Oct 27, 2023 •

edited

Loading

cellistigs commented Oct 27, 2023 •

edited

Loading