Skip to content

Training not resuming from gaussnewtonhistory #252

@geotaylo

Description

@geotaylo

I've been trying to resume a (main branch) training process that has crashed after 30 iterations (looks like it stopped part way through error fitting). However, I get this error message:

resuming from output directory output
Problem while initializing parameter c_10003 from previous training
Exception raised during training
Traceback (most recent call last):
File "/project2/rkessler/PRODUCTS/miniconda/lib/python3.7/site-packages/saltshaker/training/TrainSALT.py", line 310, in initialParameters
guess[parlist == key] = pars[names == key]
TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions

Full error log is on Midway: $DES_USERS/gtaylor/SALT3_TRAINING/intrinsic_scatter_w_sims/BS20_with_lcfit/train_screen_2.log

Command to recreate error is:

cd $DES_USERS/gtaylor/SALT3_TRAINING/intrinsic_scatter_w_sims/BS20_with_lcfit
trainsalt -c train_BS20.config --outputdir output_2 >& train_screen_2.log &

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions