New metric: loss function

[bnord]

Loss function as function of epoch (or batch)

1. top panel: Loss and validation data
2. bottom panel: residual between loss and validation curve; include dashed zero line

[bnord]

We'll probably need to specify the form of how the neural network (or whatever model) will export the loss data to be used by your module in deep diagnostics.

[bnord]

Here's an example of what is usually most helpful for formatting

[voetberg]

Refer to this notebook to get the training history: https://github.com/deepskies/DeepDiagnostics/blob/main/notebooks/train_SBI.ipynb

You'll need to figure out how to add the val loss and train loss to the saved pkl and add docs on how that's saved

[namrathaurs]

Typically, when a model is available as a .pkl, only the architecture and parameters of the model are saved in the pickle file. This is because the model (.pkl) is intended for model deployment and inference only but not for recreating the model's training process (including the associated metrics). Therefore, the model's pickle file does not inherently contain any information related to the training itself, such as training loss history, validation loss history or other training-specific metrics.

The training history can be saved separately and pickled and requires:

• saving the training history during training: involves logging loss values and other metrics at each epoch or iteration.
• storing the training history separately: which can be loaded and visualized once the model has finished training

Depending on the package being used to train the NPE models, we would have to rely on the user to capture the training summary when they train the model and save it in a format that is compatible with deepdiagnostics package to read in the training history and visualize the training/validation loss.

Questions:

1. are we only looking at SBI models that have been trained using the sbi toolkit? -- no. regardless of how the model is trained, the API should provide the capability to take the saved training summary/history data as the input and generate the visualization based on the input provided
2. should we only look at .pkl as the file format to save the training history? (.csv and .json are other options for file formats to save the training history) -- gather inputs from Roy Zhao as Brian thinks there could be a possibility to reuse Roy's code for running diagnostics.

[bnord]

Hi @rzhaolx!

[rzhaolx]

Hi Everyone! Thank you for looping me in on this :)

If I can offer my two cents (I also don't know the aim of DeepDiagnostics very well, so apologies if it's not sensible!), the loss history reader/visualizer will benefit from being very "laid back," in the sense that we could simply ask the user to supply the training and validation loss history, as simple as numpy arrays or dictionaries?

The reason for that suggestion is that, in my experience, there really isn't a uniform way to save the loss history, so only providing a pickle format reader can make the pipeline less flexible. The inference object in sbi does it in a dictionary, but if the user writes their own training loop (say with batch training or decreasing learning rate, which inference.train() does not provide), then the user has to save their own loss.

When using the inference object, I simply pickled the object, and when implementing my own training loop, I had to save them in dictionaries at the end of each epoch. I can happily supply code for this, or any other diagnostics that can be helpful :)

[voetberg]

Hi Rachel! Thanks for the input. In that case I think a better design philosophy for how we want to integrate this into the package would have a class set up like:

class LossPlot(Display): 
   def __init__(self, model=None, data=None): 
      ...

   def data_setup(self): 
      raise NotImplementedError

   def plot(
         self, display: dict = {"train_loss":....., "val_loss":....}, ...
): 
      ...
   def __call__(self, **kwargs): 
       raise NotImplementedError("Plotting loss is not supported in pipeline mode")