Poor Training Performance from nnTraining2

[jones139]

The nnTraining2 data processing and model training pipeline is producing some very poor results. There are two main issues:

1. Reproducibility is very poor - re-training a model with the same parameters can result in very different results - this suggests that the model training is unstable.
2. Sometimes we see very good training and validation results (>90% accuracy and <0.2 loss for both), but the test results are much, much worse - this suggests there is something not right in the data processing pipeline - the model is over-fitted, but the vaidation data set is not telling us that during training.

More details to follow.....

[jones139]

Reproducibility

We were initialising the random number generator for the test/train split, but if we re-run the training I keep the test/train data so the random number generator does not get initialised for subsequent runs.
This means that augmentation and random oversampling will be different on subsequent runs.

The fix of the direct cause is therefore to initialise the random number generator every time we run runSequence.py....but this does not address the underlying issue of the training being unstable.

Good Training and Validation Results, but Poor Test Results

I think what is happening here is that we are very careful to do the test/train split by event, so that data for a particular seizure only appears in either the test or train dataset.

But we are using the Keras facilty to do its own train/validate dataset split. This will do the split at the datapoint level so it is possible that data for a seizure appears in both the train and the validation datasets.

Plus we are applying data augmentation and class balancing by random oversampling to the training dataset before the validation data is separated out, so the validation data is augmented - I don't think this is good practice?

So a couple of things to fix and see what that does. Unfortunately I think this will just tell me that the model is overfitted during training, rather than having to wait for training to complete and find out when we run the test dataset on it!