Launch script failing

[alexraju91]

I got the following error while runnig the launch script.

INFO:__main__:collecting AWS resources
INFO:__main__:Creating security groups
INFO:__main__:Creating worker AMI
. None
. None
. None
. None
. None
. None
. None
. None
. None
. None
ERROR:__main__:hit max retries on <bound method ClientCreator._create_api_method.<locals>._api_call of <botocore.client.EC2 object at 0x7ff05ff16ac8>>
Traceback (most recent call last):
  File "./launch_cluster/launch.py", line 399, in <module>
    launch_manager(config)
  File "./launch_cluster/launch.py", line 57, in launch_manager
    worker_ami_id = make_worker_ami(config, ec2, [manager_security_group.id, manager_security_group2.id])
  File "./launch_cluster/launch.py", line 169, in make_worker_ami
    instance = launch_server(config, ec2, security_group_list, size=int(config.worker_ebs_size))
  File "./launch_cluster/launch.py", line 321, in launch_server
    BlockDeviceMappings=[boot_drive]
  File "./launch_cluster/launch.py", line 382, in retry
    raise e

[alexraju91]

So, the EC2 instance creation was obviously failing due to some reason. But, there was another error in the retry function. I fixed by modiying it as below:

def retry(function, *args, **kwargs):
    """ Retries a function up to max_retries, waiting `timeout` seconds between tries.
        This function is designed to retry both boto3 and fabric calls.  In the
        case of boto3, it is necessary because sometimes aws calls return too
        early and a resource needed by the next call is not yet available. """
    max_retries = kwargs.pop("max_retries", 10)
    timeout = kwargs.pop("timeout", 3)
    exception = None
    for i in range(max_retries):
        print (".", sys.stdout.flush())
        try:
            return function(*args, **kwargs)
        except (ClientError, NetworkError, WaiterError) as e:
            logger.debug("retrying %s, (~%s seconds elapsed)" % (function, i * 3))
            sleep(timeout)
            exception = e

    logger.error("hit max retries on %s" % function)
    raise exception

[jakobmaier-eu]

Hi Alex,
Concerning the first error, I suppose you did not include the attribute "--custom_worker_ami ami-f3fa8de5" when running the launch.py script. This lead to errors in my case as well. Did you fix it by including the attribute?

Just like in your case, I encountered the second Error during the "INFO:main:Launching manager instance" -Step:

Ignoring ssh key permissions
INFO:__main__:collecting AWS resources
INFO:__main__:Creating security groups
INFO:__main__:Custom worker AMI id 'ami-f3fa8de5' specified. Using custom AMI to launch worker user servers.
INFO:__main__:Launching manager instance
. None
. None
. None
. None
. None
. None
. None
. None
. None
. None
ERROR:__main__:hit max retries on <bound method ClientCreator._create_api_method.<locals>._api_call of <botocore.client.EC2 object at 0x7f35103b3588>>
Traceback (most recent call last):
  File "./launch_cluster/launch.py", line 399, in <module>
    launch_manager(config)
  File "./launch_cluster/launch.py", line 64, in launch_manager
    instance = launch_server(config, ec2, [manager_security_group.id, manager_security_group2.id])
  File "./launch_cluster/launch.py", line 321, in launch_server
    BlockDeviceMappings=[boot_drive]
  File "./launch_cluster/launch.py", line 382, in retry
    raise e
UnboundLocalError: local variable 'e' referenced before assignment

According to https://www.toptal.com/python/top-10-mistakes-that-python-programmers-make , see Mistake 9 on this website , this seems to be a migration issue from Python2 to Python3.
I am wondering whether this has been overseen? @farassadek

[alexraju91]

Hi @jakobmaier-eu
I didn't set the Key path properly. That was the real issue. #8

After fixing this, the launch script is working properly. I didn't set the custom_worker_ami attribute.

[jakobmaier-eu]

@alexforever86 I see. But the error you fixed in your second comment in this Issue-Thread is still present is it not?

[farassadek]

Is this issue solved by fixing the key path ?

[alexraju91]

@farassadek This thread is addressing the issue with the try exception. If the function passed to retry method is buggy, local variable 'e' referenced before assignment error appears. LIke @jakobmaier-eu said, this is a migration issue from python 2 to 3.

[farassadek]

This variable should be defined before called . I will fix it.