Your GCP script

[IEWbgfnYDwHRoRRSKtkdyMDUzgdwuBYgDKtDJWd]

First of all, much thanks and appreciation for your repo, your script for GCP setup worked like a charm.

Only issue is when I try to train a new model using a custom dataset, it errors about 20 minutes after the first tick. also seems to have initial sample fake outputs as human faces (my dataset isnt faces). Unsure if this is normal or if I am doing something wrong.

[dvschultz]

can you post the error you‘re getting.

Seeing faces for your first fake is correct. This is using transfer learning (you can look it up on my youtube page and learn more about the technique there). Those faces get erased completely after a handful of ticks.

[ahmedshingaly]

Thank you very much for the great tutorial
my GPU is running out of memory and failing in the beginning, appreciate if you take a look at the error bellow

her is my error
`2020-05-20 11:18:44.904340: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 36.00MiB (rounded to 37748736). Current allocation summary follows.
2020-05-20 11:18:44.914558: W tensorflow/core/common_runtime/bfc_allocator.cc:319] **********************************************************************************************xx
2020-05-20 11:18:44.919358: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,4,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
return fn(*args)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,3,3,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node GPU0/G_loss/G/G_synthesis/128x128/Conv1/Square}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_training.py", line 192, in
main()
File "run_training.py", line 187, in main
run(**vars(args))
File "run_training.py", line 120, in run
dnnlib.submit_run(**kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\internal\local.py", line 22, in submit
return run_wrapper(submit_config)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\training\training_loop.py", line 299, in training_loop
tflib.run(G_train_op, feed_dict)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\tflib\tfutil.py", line 31, in run
return tf.get_default_session().run(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
run_metadata)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,3,3,128,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node GPU0/G_loss/G/G_synthesis/128x128/Conv1/Square (defined at C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py:104) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Errors may have originated from an input operation.
Input Source operations connected to node GPU0/G_loss/G/G_synthesis/128x128/Conv1/Square:
GPU0/G_loss/G/G_synthesis/128x128/Conv1/mul_3 (defined at C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py:100)

Original stack trace for 'GPU0/G_loss/G/G_synthesis/128x128/Conv1/Square':
File "run_training.py", line 192, in
main()
File "run_training.py", line 187, in main
run(**vars(args))
File "run_training.py", line 120, in run
dnnlib.submit_run(**kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\internal\local.py", line 22, in submit
return run_wrapper(submit_config)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\submission\submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\training\training_loop.py", line 220, in training_loop
G_loss, G_reg = dnnlib.util.call_func_by_name(G=G_gpu, D=D_gpu, opt=G_opt, training_set=training_set, minibatch_size=minibatch_gpu_in, **G_loss_args)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\util.py", line 256, in call_func_by_name
return func_obj(*args, **kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\training\loss.py", line 152, in G_logistic_ns_pathreg
fake_images_out, fake_dlatents_out = G.get_output_for(latents, labels, is_training=True, return_dlatents=True)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\tflib\network.py", line 221, in get_output_for
out_expr = self._build_func(*final_inputs, **build_kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py", line 238, in G_main
images_out = components.synthesis.get_output_for(dlatents, is_training=is_training, force_clean_graph=is_template_graph, **kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\dnnlib\tflib\network.py", line 221, in get_output_for
out_expr = self._build_func(final_inputs, **build_kwargs)
File "C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py", line 498, in G_synthesis_stylegan2
x = block(x, res)
File "C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py", line 470, in block
x = layer(x, layer_idx=res2-4, fmaps=nf(res-1), kernel=3)
File "C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py", line 455, in layer
x = modulated_conv2d_layer(x, dlatents_in[:, layer_idx], fmaps=fmaps, kernel=kernel, up=up, resample_kernel=resample_kernel, fused_modconv=fused_modconv)
File "C:\Users\USER6459\Documents\python\stylegan2\training\networks_stylegan2.py", line 104, in modulated_conv2d_layer
d = tf.rsqrt(tf.reduce_sum(tf.square(ww), axis=[1,2,3]) + 1e-8) # [BO] Scaling factor.
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 10698, in square
"Square", x=x, name=name)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
op_def=op_def)
File "C:\ProgramData\Anaconda3\envs\old_tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()`

[dvschultz]

can you tell me what GPU you’re using, the CUDA version you’re running and what NVIDIA driver is running?

[ahmedshingaly]

I use workstation RTX 2080 Ti 11GB X1 and Ram is 128GM
CUDA version is 10.0
and even if I reduce batch size it still fails due to space lack
my dataset is 500 pictures 1024x1024
i tried bigger data set and smaller dataset and tried png, jpg,
I tried all config avilable (a , b , c ,d ,e , f)
I tried running it on another computer with GTX 1060 and cuda fails
I tried 512 by 512 image dataset
I tried 256 x 256 image dataset

< ERROR CUDA RUN OUT OF MEMORY >
please note, I am trying to build my own models on building shapes not human faces
thank you in advance