I failed to reproduce SpliceAI training based on the source code. #155
Description
Hello. Great work. I am trying to reproduce SpliceAI model from the source code you provided.
Following the README.txt instruction, I started training, but I found the training was only valid when the context of 80 bp.
I mean when I increase the context to 400~10,000bp, the validation metrics become almost zero.
The installed softwere version is here. And further below, I added the output log by $ python -u train_model.py 2000 1.
Could you tell me advices to successfully reproduce the training?
$ conda list
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
_tflow_select 2.1.0 gpu
absl-py 0.15.0 pyhd3eb1b0_0
astor 0.8.0 py27_0
backports 1.1 pyhd3eb1b0_1
backports.weakref 1.0.post1 py_1
blas 1.1 openblas conda-forge
c-ares 1.19.1 h5eee18b_0
ca-certificates 2024.12.31 h06a4308_0
certifi 2020.6.20 pyhd3eb1b0_3
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cupti 10.0.130 0
enum34 1.1.6 py27_1
funcsigs 1.0.2 py27_0
future 0.18.2 py27_0
futures 3.3.0 py27_0
gast 0.2.2 py27_0
google-pasta 0.2.0 pyhd3eb1b0_0
grpcio 1.12.1 py27hdbcaa40_0
h5py 2.9.0 py27h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
intel-openmp 2023.1.0 hdb19cb5_46306
keras 2.2.4 0
keras-applications 1.0.8 py_1
keras-base 2.2.4 py27_0
keras-preprocessing 1.1.0 py_1
libblas 3.9.0 1_h86c2bf4_netlib conda-forge
libcblas 3.9.0 6_ha36c22a_netlib conda-forge
libffi 3.4.4 h6a678d5_1
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libgfortran5 14.2.0 hd5240d6_1 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
liblapack 3.9.0 6_ha36c22a_netlib conda-forge
libopenblas 0.2.20 h9ac9557_7
libprotobuf 3.11.2 hd408876_0
libstdcxx-ng 11.2.0 h1234567_1
linecache2 1.0.0 py_1
markdown 3.1.1 py27_0
mkl 2020.2 256
mkl-service 2.3.0 py27he904b0f_0
mkl_fft 1.0.10 py27_0 conda-forge
mkl_random 1.0.2 py27_0 conda-forge
mock 2.0.0 py27_0
ncurses 6.4 h6a678d5_0
numpy 1.16.5 py27h95a1406_0 conda-forge
numpy-base 1.14.3 py27h2b20989_0
openblas 0.3.3 h9ac9557_1001 conda-forge
openssl 3.4.0 h7b32b05_1 conda-forge
opt_einsum 2.3.2 py_0
pbr 5.5.1 py_0
pip 19.3.1 py27_0
protobuf 3.11.2 py27he6710b0_0
python 2.7.18 h42bf7aa_3
pyyaml 5.2 py27h7b6447c_0
readline 8.2 h5eee18b_0
scikit-learn 0.20.4 py27_blas_openblashebff5e3_0 conda-forge
scipy 1.2.1 py27h921218d_2 conda-forge
setuptools 44.0.0 py27_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.45.3 h5eee18b_0
tensorboard 1.14.0 py27_0 conda-forge
tensorflow 1.14.0 gpu_py27he9627f8_0
tensorflow-base 1.14.0 gpu_py27h8d69cac_0
tensorflow-estimator 1.14.0 py27h5ca1d4c_0 conda-forge
tensorflow-gpu 1.14.0 h0d30ee6_0
termcolor 1.1.0 py27_1
tk 8.6.14 h39e8969_0
traceback2 1.4.0 py_0
unittest2 1.1.0 pyhd3eb1b0_0
webencodings 0.5.1 py27_1
werkzeug 0.16.1 py_0
wheel 0.37.1 pyhd3eb1b0_0
wrapt 1.11.2 py27h7b6447c_0
yaml 0.1.7 had09818_2
zlib 1.2.13 h5eee18b_1
$ python -u train_model.py 2000 1
Using TensorFlow backend.
Context nucleotides: 2000
Sequence length (output): 5000
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, None, 4) 0
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, None, 32) 160 input_1[0][0]
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, None, 32) 128 conv1d_1[0][0]
__________________________________________________________________________________________________
activation_1 (Activation) (None, None, 32) 0 batch_normalization_1[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, None, 32) 11296 activation_1[0][0]
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, None, 32) 128 conv1d_3[0][0]
__________________________________________________________________________________________________
activation_2 (Activation) (None, None, 32) 0 batch_normalization_2[0][0]
__________________________________________________________________________________________________
conv1d_4 (Conv1D) (None, None, 32) 11296 activation_2[0][0]
__________________________________________________________________________________________________
add_1 (Add) (None, None, 32) 0 conv1d_4[0][0]
conv1d_1[0][0]
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, None, 32) 128 add_1[0][0]
__________________________________________________________________________________________________
activation_3 (Activation) (None, None, 32) 0 batch_normalization_3[0][0]
__________________________________________________________________________________________________
conv1d_5 (Conv1D) (None, None, 32) 11296 activation_3[0][0]
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, None, 32) 128 conv1d_5[0][0]
__________________________________________________________________________________________________
activation_4 (Activation) (None, None, 32) 0 batch_normalization_4[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D) (None, None, 32) 11296 activation_4[0][0]
__________________________________________________________________________________________________
add_2 (Add) (None, None, 32) 0 conv1d_6[0][0]
add_1[0][0]
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, None, 32) 128 add_2[0][0]
__________________________________________________________________________________________________
activation_5 (Activation) (None, None, 32) 0 batch_normalization_5[0][0]
__________________________________________________________________________________________________
conv1d_7 (Conv1D) (None, None, 32) 11296 activation_5[0][0]
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, None, 32) 128 conv1d_7[0][0]
__________________________________________________________________________________________________
activation_6 (Activation) (None, None, 32) 0 batch_normalization_6[0][0]
__________________________________________________________________________________________________
conv1d_8 (Conv1D) (None, None, 32) 11296 activation_6[0][0]
__________________________________________________________________________________________________
add_3 (Add) (None, None, 32) 0 conv1d_8[0][0]
add_2[0][0]
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, None, 32) 128 add_3[0][0]
__________________________________________________________________________________________________
activation_7 (Activation) (None, None, 32) 0 batch_normalization_7[0][0]
__________________________________________________________________________________________________
conv1d_9 (Conv1D) (None, None, 32) 11296 activation_7[0][0]
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, None, 32) 128 conv1d_9[0][0]
__________________________________________________________________________________________________
activation_8 (Activation) (None, None, 32) 0 batch_normalization_8[0][0]
__________________________________________________________________________________________________
conv1d_10 (Conv1D) (None, None, 32) 11296 activation_8[0][0]
__________________________________________________________________________________________________
add_4 (Add) (None, None, 32) 0 conv1d_10[0][0]
add_3[0][0]
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, None, 32) 128 add_4[0][0]
__________________________________________________________________________________________________
activation_9 (Activation) (None, None, 32) 0 batch_normalization_9[0][0]
__________________________________________________________________________________________________
conv1d_12 (Conv1D) (None, None, 32) 11296 activation_9[0][0]
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, None, 32) 128 conv1d_12[0][0]
__________________________________________________________________________________________________
activation_10 (Activation) (None, None, 32) 0 batch_normalization_10[0][0]
__________________________________________________________________________________________________
conv1d_13 (Conv1D) (None, None, 32) 11296 activation_10[0][0]
__________________________________________________________________________________________________
add_6 (Add) (None, None, 32) 0 conv1d_13[0][0]
add_4[0][0]
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, None, 32) 128 add_6[0][0]
__________________________________________________________________________________________________
activation_11 (Activation) (None, None, 32) 0 batch_normalization_11[0][0]
__________________________________________________________________________________________________
conv1d_14 (Conv1D) (None, None, 32) 11296 activation_11[0][0]
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, None, 32) 128 conv1d_14[0][0]
__________________________________________________________________________________________________
activation_12 (Activation) (None, None, 32) 0 batch_normalization_12[0][0]
__________________________________________________________________________________________________
conv1d_15 (Conv1D) (None, None, 32) 11296 activation_12[0][0]
__________________________________________________________________________________________________
add_7 (Add) (None, None, 32) 0 conv1d_15[0][0]
add_6[0][0]
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, None, 32) 128 add_7[0][0]
__________________________________________________________________________________________________
activation_13 (Activation) (None, None, 32) 0 batch_normalization_13[0][0]
__________________________________________________________________________________________________
conv1d_16 (Conv1D) (None, None, 32) 11296 activation_13[0][0]
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, None, 32) 128 conv1d_16[0][0]
__________________________________________________________________________________________________
activation_14 (Activation) (None, None, 32) 0 batch_normalization_14[0][0]
__________________________________________________________________________________________________
conv1d_17 (Conv1D) (None, None, 32) 11296 activation_14[0][0]
__________________________________________________________________________________________________
add_8 (Add) (None, None, 32) 0 conv1d_17[0][0]
add_7[0][0]
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, None, 32) 128 add_8[0][0]
__________________________________________________________________________________________________
activation_15 (Activation) (None, None, 32) 0 batch_normalization_15[0][0]
__________________________________________________________________________________________________
conv1d_18 (Conv1D) (None, None, 32) 11296 activation_15[0][0]
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, None, 32) 128 conv1d_18[0][0]
__________________________________________________________________________________________________
activation_16 (Activation) (None, None, 32) 0 batch_normalization_16[0][0]
__________________________________________________________________________________________________
conv1d_19 (Conv1D) (None, None, 32) 11296 activation_16[0][0]
__________________________________________________________________________________________________
add_9 (Add) (None, None, 32) 0 conv1d_19[0][0]
add_8[0][0]
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, None, 32) 128 add_9[0][0]
__________________________________________________________________________________________________
activation_17 (Activation) (None, None, 32) 0 batch_normalization_17[0][0]
__________________________________________________________________________________________________
conv1d_21 (Conv1D) (None, None, 32) 21536 activation_17[0][0]
__________________________________________________________________________________________________
batch_normalization_18 (BatchNo (None, None, 32) 128 conv1d_21[0][0]
__________________________________________________________________________________________________
activation_18 (Activation) (None, None, 32) 0 batch_normalization_18[0][0]
__________________________________________________________________________________________________
conv1d_22 (Conv1D) (None, None, 32) 21536 activation_18[0][0]
__________________________________________________________________________________________________
add_11 (Add) (None, None, 32) 0 conv1d_22[0][0]
add_9[0][0]
__________________________________________________________________________________________________
batch_normalization_19 (BatchNo (None, None, 32) 128 add_11[0][0]
__________________________________________________________________________________________________
activation_19 (Activation) (None, None, 32) 0 batch_normalization_19[0][0]
__________________________________________________________________________________________________
conv1d_23 (Conv1D) (None, None, 32) 21536 activation_19[0][0]
__________________________________________________________________________________________________
batch_normalization_20 (BatchNo (None, None, 32) 128 conv1d_23[0][0]
__________________________________________________________________________________________________
activation_20 (Activation) (None, None, 32) 0 batch_normalization_20[0][0]
__________________________________________________________________________________________________
conv1d_24 (Conv1D) (None, None, 32) 21536 activation_20[0][0]
__________________________________________________________________________________________________
add_12 (Add) (None, None, 32) 0 conv1d_24[0][0]
add_11[0][0]
__________________________________________________________________________________________________
batch_normalization_21 (BatchNo (None, None, 32) 128 add_12[0][0]
__________________________________________________________________________________________________
activation_21 (Activation) (None, None, 32) 0 batch_normalization_21[0][0]
__________________________________________________________________________________________________
conv1d_25 (Conv1D) (None, None, 32) 21536 activation_21[0][0]
__________________________________________________________________________________________________
batch_normalization_22 (BatchNo (None, None, 32) 128 conv1d_25[0][0]
__________________________________________________________________________________________________
activation_22 (Activation) (None, None, 32) 0 batch_normalization_22[0][0]
__________________________________________________________________________________________________
conv1d_26 (Conv1D) (None, None, 32) 21536 activation_22[0][0]
__________________________________________________________________________________________________
add_13 (Add) (None, None, 32) 0 conv1d_26[0][0]
add_12[0][0]
__________________________________________________________________________________________________
batch_normalization_23 (BatchNo (None, None, 32) 128 add_13[0][0]
__________________________________________________________________________________________________
activation_23 (Activation) (None, None, 32) 0 batch_normalization_23[0][0]
__________________________________________________________________________________________________
conv1d_27 (Conv1D) (None, None, 32) 21536 activation_23[0][0]
__________________________________________________________________________________________________
batch_normalization_24 (BatchNo (None, None, 32) 128 conv1d_27[0][0]
__________________________________________________________________________________________________
activation_24 (Activation) (None, None, 32) 0 batch_normalization_24[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, None, 32) 1056 conv1d_1[0][0]
__________________________________________________________________________________________________
conv1d_11 (Conv1D) (None, None, 32) 1056 add_4[0][0]
__________________________________________________________________________________________________
conv1d_28 (Conv1D) (None, None, 32) 21536 activation_24[0][0]
__________________________________________________________________________________________________
add_5 (Add) (None, None, 32) 0 conv1d_2[0][0]
conv1d_11[0][0]
__________________________________________________________________________________________________
conv1d_20 (Conv1D) (None, None, 32) 1056 add_9[0][0]
__________________________________________________________________________________________________
add_14 (Add) (None, None, 32) 0 conv1d_28[0][0]
add_13[0][0]
__________________________________________________________________________________________________
add_10 (Add) (None, None, 32) 0 add_5[0][0]
conv1d_20[0][0]
__________________________________________________________________________________________________
conv1d_29 (Conv1D) (None, None, 32) 1056 add_14[0][0]
__________________________________________________________________________________________________
add_15 (Add) (None, None, 32) 0 add_10[0][0]
conv1d_29[0][0]
__________________________________________________________________________________________________
cropping1d_1 (Cropping1D) (None, None, 32) 0 add_15[0][0]
__________________________________________________________________________________________________
conv1d_30 (Conv1D) (None, None, 3) 99 cropping1d_1[0][0]
==================================================================================================
Total params: 360,579
Trainable params: 359,043
Non-trainable params: 1,536
__________________________________________________________________________________________________
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
WARNING:tensorflow:From /home/shiro/miniconda3/envs/keras-py2-gpu/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1521: The name tf.log is deprecated. Please use tf.math.log instead.
('BATCH_SIZE:', 24)
2025-01-29 14:47:09.947963: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2025-01-29 14:47:09.955182: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2304005000 Hz
2025-01-29 14:47:09.956888: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xed23240 executing computations on platform Host. Devices:
2025-01-29 14:47:09.956935: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2025-01-29 14:47:09.960022: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2025-01-29 14:47:10.349228: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.349398: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9dae4b0 executing computations on platform CUDA. Devices:
2025-01-29 14:47:10.349430: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA GeForce RTX 3070 Laptop GPU, Compute Capability 8.6
2025-01-29 14:47:10.349676: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.349745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA GeForce RTX 3070 Laptop GPU major: 8 minor: 6 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2025-01-29 14:47:10.352229: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2025-01-29 14:47:10.368687: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2025-01-29 14:47:10.377357: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2025-01-29 14:47:10.380307: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2025-01-29 14:47:10.400169: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2025-01-29 14:47:10.412138: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2025-01-29 14:47:10.444871: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2025-01-29 14:47:10.445058: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445105: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2025-01-29 14:47:10.445220: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2025-01-29 14:47:10.445651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-01-29 14:47:10.445675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2025-01-29 14:47:10.445684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2025-01-29 14:47:10.445938: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.445959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1409] Could not identify NUMA node of platform GPU id 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2025-01-29 14:47:10.445984: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2025-01-29 14:47:10.446045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6762 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3070 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6)
2025-01-29 14:47:21.554931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
--------------------------------------------------------------
Validation set metrics:
Acceptor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000 0.0000 0.0001 0.0004 0.0002 0.0000 0.0000 0.0000 0.0000 12893
Donor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000 0.0000 0.0001 0.0004 0.0002 0.0000 0.0000 0.0000 0.0000 12927
Training set metrics:
Acceptor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000 0.0001 0.0005 0.0008 0.0001 0.0000 0.0000 0.0000 0.0000 13117
Donor:
Warning: y_pred contains NaN or infinity. Converting to 0.
0.0000 0.0000 0.0004 0.0008 0.0001 0.0000 0.0000 0.0000 0.0000 13141
Learning rate: 0.00100
--- 6627.18091917 seconds ---