Unable to train joint model from pretrained cpg and dna model with tensorflow backend

[snajder-r]

Hi,

I was able to train the joint model successfully. Then I wanted to train the cpg and dna model separately and the joint model on top. Training cpg and dna model worked fine, but I started having problems when trying to train the joint model from the two pretrained models.

The error I got was:

tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value cpg/bidirectional_1/forward_gru_1/bias
	 [[{{node cpg/bidirectional_1/forward_gru_1/bias/read}}]]

I found that the cause was in dcpg_train.py in get_callbacks:

if K._BACKEND == 'tensorflow' and not opts.no_tensorboard:
    K.set_session(K.tf.Session(config=K.tf.ConfigProto(
        intra_op_parallelism_threads=1,
        inter_op_parallelism_threads=1)))

I don't have too much experience with tensorflow or Keras, but it looks to me that setting a new session causes the model to lose the loaded weights. Since this is only done when tensorboard is used, I was able to workaround the problem by running with --no_tensorboard.

Best,
Rene

[cangermueller]

Thanks for letting me know. I will try to fix this. For now, I suggest running without tensorboard.

…

On Fri, Aug 30, 2019 at 2:03 AM snajder-r ***@***.***> wrote: Hi, I was able to train the joint model successfully. Then I wanted to train the cpg and dna model separately and the joint model on top. Training cpg and dna model worked fine, but I started having problems when trying to train the joint model from the two pretrained models. The error I got was: tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value cpg/bidirectional_1/forward_gru_1/bias [[{{node cpg/bidirectional_1/forward_gru_1/bias/read}}]] I found that the cause was in dcpg_train.py in get_callbacks: if K._BACKEND == 'tensorflow' and not opts.no_tensorboard: K.set_session(K.tf.Session(config=K.tf.ConfigProto( intra_op_parallelism_threads=1, inter_op_parallelism_threads=1))) I don't have too much experience with tensorflow or Keras, but it looks to me that setting a new session causes the model to lose the loaded weights. Since this is only done when tensorboard is used, I was able to workaround the problem by running with --no_tensorboard. Best, Rene — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ABEVZM75L7HVDQ3FGFPPXQLQHDO5DA5CNFSM4ISLBZC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HIMWZLQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEVZMZ6WY7NIKEOSIODTDLQHDO5DANCNFSM4ISLBZCQ> .

-- Christof Angermueller cangermueller@gmail.com https://cangermueller.com