-
Notifications
You must be signed in to change notification settings - Fork 1
Description
I've been trying to resume a (main branch) training process that has crashed after 30 iterations (looks like it stopped part way through error fitting). However, I get this error message:
resuming from output directory output
Problem while initializing parameter c_10003 from previous training
Exception raised during training
Traceback (most recent call last):
File "/project2/rkessler/PRODUCTS/miniconda/lib/python3.7/site-packages/saltshaker/training/TrainSALT.py", line 310, in initialParameters
guess[parlist == key] = pars[names == key]
TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions
Full error log is on Midway: $DES_USERS/gtaylor/SALT3_TRAINING/intrinsic_scatter_w_sims/BS20_with_lcfit/train_screen_2.log
Command to recreate error is:
cd $DES_USERS/gtaylor/SALT3_TRAINING/intrinsic_scatter_w_sims/BS20_with_lcfit
trainsalt -c train_BS20.config --outputdir output_2 >& train_screen_2.log &