Configure Delegates on AWS Solution

[ibr123]

Hi,

i want to create a master Delegate and another monitor Delegate, but the this is that i have access to AWS CLI and i have permission to manipulate the system and create any required service.
so what are the steps to follow the installation of Delegates but without using the "ho" since i can configure everything from the CLI? meaning what are the steps to do assuming the AWS environment is already set?

Thanks

[smithfarm]

@ibr123 This project is something I use occasionally to create demo Ceph clusters in AWS.

After running "ho" to create the master and the delegates, I ssh to the master. From there, I use Salt/DeepSea to deploy Ceph clusters on the delegates, but that part (deployment) is more-or-less beyond the scope of this project.

[ibr123]

Hi,
I got your point, but what is happening is that I followed the steps that you already shared, which is creating YAML, installing python-virtualenv, configuring roles....etc. the steps that requires me to create AWS components I have done that already through the console management , validation YAML is fine, but at the step ho install keypairs --all --master I get the following error:

(virtualenv) *******:~/ceph_app/ceph-auto-aws$ ho install delegates --master
2018-02-11 07:58:08,188 INFO Loaded yaml tree from './aws.yaml'
2018-02-11 07:58:08,188 INFO Delegate list is [0]
2018-02-11 07:58:08,188 INFO Installing cluster for delegate 0
2018-02-11 07:58:08,250 INFO Keypair object ubuntu-d0 fetched from AWS
2018-02-11 07:58:08,701 ERROR 400 Bad Request
2018-02-11 07:58:08,701 ERROR <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSubnet.Conflict</Code><Message>The CIDR '10.0.0.0/24' conflicts with another subnet</Message></Error></Errors><RequestID>047d6cd5-33a5-41bd-a29c-970962398849</RequestID></Response>
Traceback (most recent call last):
  File "/home/ubuntu/ceph_app/ceph-auto-aws/virtualenv/bin/ho", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/ubuntu/ceph_app/ceph-auto-aws/bin/ho", line 24, in <module>
    sys.exit(HandsOn().run(sys.argv[1:]))
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/main.py", line 195, in run
    self.args.func(self.args).run()
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/install.py", line 185, in run
    d = Delegate(self.args, d)
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/delegate.py", line 57, in __init__
    s_obj = s.subnet_obj(create=True, dry_run=self.args.dry_run)
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/subnet.py", line 80, in subnet_obj
    availability_zone=az
  File "/home/ubuntu/ceph_app/ceph-auto-aws/virtualenv/local/lib/python2.7/site-packages/boto/vpc/__init__.py", line 1180, in create_subnet
    return self.get_object('CreateSubnet', params, Subnet)
  File "/home/ubuntu/ceph_app/ceph-auto-aws/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidSubnet.Conflict</Code><Message>The CIDR '10.0.0.0/24' conflicts with another subnet</Message></Error></Errors><RequestID>047d6cd5-33a5-41bd-a29c-970962398849</RequestID></Response>

keep in mind I created the subnets using Console Management not from the command: ho install subnets --all --master
since the subnet is already created, when I try to install using ho install delegates --master why its trying to create subnets again? -as you can see from the error above its trying to create a subnet that its already there-
by the way, in my aws.yaml subnet section look like: subnets: {} I should add the subnet manually? if yes, how it should look like? considering I only have two delegates, one is the master and the other is a monitor

Thanks

UPADTE: even though I have added subnets to the aws.yaml manually, i still have the same error

to check the yaml file i did: ho probe yaml and gave me

~/ceph_app/ceph-auto-aws$ ho probe yaml
2018-02-11 10:05:55,171 INFO Probing 'subnets' stanza
2018-02-11 10:05:55,178 INFO Loaded yaml tree from './aws.yaml'
2018-02-11 10:05:55,178 INFO Probing 'keypairs' stanza
2018-02-11 10:05:55,178 INFO Probing 'cluster-definition' stanza
2018-02-11 10:05:55,178 INFO Detected cluster-definition stanza
2018-02-11 10:05:55,178 INFO Detected role 'admin' in cluster definition
2018-02-11 10:05:55,178 INFO Detected role 'mon1' in cluster definition
2018-02-11 10:05:55,178 INFO Probing 'nametag' stanza
2018-02-11 10:05:55,178 INFO Probing 'delegates' stanza
2018-02-11 10:05:55,178 INFO Probing 'vpc' stanza
2018-02-11 10:05:55,178 INFO Probing 'types' stanza
2018-02-11 10:05:55,178 INFO Probing 'role-definitions' stanza
2018-02-11 10:05:55,179 INFO Detected roles ['admin', 'windows', 'mon2', 'mon3', 'defaults', 'mon1', 'master', 'osd']
2018-02-11 10:05:55,179 INFO Probing 'region' stanza
2018-02-11 10:05:55,179 INFO Probing 'keyname' stanza
2018-02-11 10:05:55,179 INFO Probing 'clusters' stanza
2018-02-11 10:05:55,179 INFO YAML tree is sane

I added the subnets like the following:

subnets:
    salt_master_subnet: '**.**.0.0/24'
    salt_delegate1_subnet: '**.**.1.0/24'

UPDATE2: I ran the command ho install delegates --master --dry-run it gave me the following error:

~/ceph_app/ceph-auto-aws$ ho install delegates --master --dry-run
2018-02-11 12:31:19,846 INFO Loaded yaml tree from './aws.yaml'
2018-02-11 12:31:19,846 INFO Delegate list is [0]
2018-02-11 12:31:19,846 INFO Installing cluster for delegate 0
2018-02-11 12:31:19,913 INFO Keypair object ubuntu-d0 fetched from AWS
2018-02-11 12:31:20,200 INFO Dry run: doing nothing
Traceback (most recent call last):
  File "/home/ubuntu/ceph_app/ceph-auto-aws/virtualenv/bin/ho", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/ubuntu/ceph_app/ceph-auto-aws/bin/ho", line 24, in <module>
    sys.exit(HandsOn().run(sys.argv[1:]))
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/main.py", line 195, in run
    self.args.func(self.args).run()
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/install.py", line 186, in run
    d.install(dry_run=self.args.dry_run)
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/delegate.py", line 219, in install
    self._delegate['roles'] = self.ready_to_install(dry_run=dry_run)
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/delegate.py", line 123, in ready_to_install
    if self.preexisting_instances():
  File "/home/ubuntu/ceph_app/ceph-auto-aws/handson/delegate.py", line 77, in preexisting_instances
    s_id = s_obj.id
AttributeError: 'NoneType' object has no attribute 'id'

[ibr123]

I ran the command ho probe subnets :

~/ceph_app/ceph-auto-aws$ ho probe subnets
2018-02-11 14:28:11,524 INFO Loaded yaml tree from './aws.yaml'
2018-02-11 14:28:11,524 INFO Probing 2 subnets
2018-02-11 14:28:11,831 INFO Delegate 0 subnet ID missing in yaml
2018-02-11 14:28:12,163 INFO Delegate 1 subnet ID missing in yaml

and this is the roles at the aws.yaml

role-definitions:
 admin:
   last-octet: 11
 defaults:
   ami-id: *******
   last-octet:
   node-no:
   replace-from-environment: []
   type: *******
   user-data: data/user-data-minions
   volume:
 master:
   last-octet: 11
   user-data: data/user-data-master
   volume:
 mon1:
   last-octet: 12
 mon2:
   last-octet: 16
 mon3:
   last-octet: 13
 osd:
   last-octet: 14
 windows:
   last-octet: 15

keep in mind that the master's IP Address end with 0.11 and the monitor's delegate, its IP Address ends with 0.12

[smithfarm]

The subnets must be filled in - subnet 0 for the master and subnets 1-n for delegates 1-n, where "n" is the number of delegates set in the YAML file by the line that says "delegates: [n]".

Each subnet stanza looks like this:

  0:
    cidr_block: 10.0.0.0/24
    id: subnet-738c363e

You can find the subnet ID from the AWS web GUI. You should also fill in the VPC stanza:

vpc:
  cidr_block: 10.0.0.0/16
  id: vpc-c29ebcab

You do have a VPC, right?

[ibr123]

yes I do have a VPC, but a question, I added the two subnets to my YAML, the id is the same for both subnets, correct?
also according to the line in the installation

Each delegate will be assigned a number, e.g. 12. The subnet of delegate 12 will be 10.0.12.0/24.

the subnet to the second delegate will be as the following: cidr_block: 10.0.1.0/24 right?

Thanks

[smithfarm]

No, you have to find the actual subnet ID and cidr block for the VPC and each subnet. The AWS dashboard contains this information.

Each delegate will be assigned a number, e.g. 12. The subnet of delegate 12 will be 10.0.12.0/24.

Since you are not using ho as it was designed, this sentence does not apply to you.

[ibr123]

ok i hear you, i have created another subnet for the delegate 1, now my aws.yaml looks like:

cluster-definition:
  - role: admin
  - role: mon1

clusters: {}

delegates: 1

keyname: *******

keypairs:
  0:
    keyname: *****-d0
  1:
    keyname: ******-d1

nametag: handson

region:
  availability_zone: ********
  region_str: us-east-1

role-definitions:
  admin:
    last-octet:
  defaults:
    ami-id: ami-********
    last-octet:
    node-no:
    replace-from-environment: []
    type: *******
    user-data: data/user-data-minions
    volume:
  master:
    last-octet: 11
    user-data: data/user-data-master
    volume:
  mon1:
    last-octet: 12
  mon2:
    last-octet: 16
  mon3:
    last-octet: 13
  osd:
    last-octet: 14
  windows:
    last-octet: 15

subnets:
  0:
    cidr_block: 10.0.0.0/24
    id: subnet-*******
  1:
    cidr_block: 10.0.1.0/24
    id: subnet-*******

types:
  - m5.large

vpc:
  cidr_block: 10.0.0.0/16
  id: vpc-*******

yet i have the sentence INFO Delegate 0 not instantiated continuously, i know its showing because the attribute "cluster" is empty in the aws.yaml, but as you already know i created AWS environment manually :) , so how its should look like in the aws.yaml?
another question, is it OK if the master and delegate 1 were in the same subnet? just wondering

[smithfarm]

I have a question for you. If you are creating the AWS environment manually, what benefit do you think you will get from using ho?

[smithfarm]

  admin:
    last-octet:

You need last-octet: 10 here.

The master must be in its own subnet - i.e. subnet id for subnet 0 must be different than for subnet 1. And you need to instantiate the master delegate as described in the documentation.

[ibr123]

Sorry for the typo, the admin last-octet: is 11, same as the masters last-octet, which is following the spin up delegate cluster so its like this

admin:
    last-octet: 11

i think the problem resides at this line of code that's why i needed to know how the 'cluster' looks like in the aws.yaml.
as for your question

If you are creating the AWS environment manually, what benefit do you think you will get from using ho?

i have no prior experience with installing Salt and DeepSea, for the steps that creates AWS components such as VPC, or subnets, i already have that, but for probing and checking the validation of VPC connection for example or probing the YAML, or even installing the delegates, there where i use ho

[smithfarm]

So what do you get when you do this step? https://github.com/smithfarm/ceph-auto-aws#install-salt-master

Can you confirm that subnet 0 and subnet 1 have different IDs now?

[smithfarm]

The "cluster" YAML stanza will be populated by ho when it successfully instantiates a delegate. Then you will see what it looks like ;-)

[ibr123]

from the command that you have referred to, i get the following:

2018-02-12 09:32:36,195 INFO Loaded yaml tree from './aws.yaml'
2018-02-12 09:32:36,195 INFO Delegate list is [0]
2018-02-12 09:32:36,195 INFO Installing cluster for delegate 0
2018-02-12 09:32:36,258 INFO Keypair object ubuntu-d0 fetched from AWS
2018-02-12 09:32:36,642 INFO Found subnet subnet-50be7f5f (10.0.0.0/24)
2018-02-12 09:32:36,801 WARNING Delegate 0 (subnet 10.0.0.0/24) already has 2 instances
2018-02-12 09:32:36,862 INFO Keypair object ubuntu-d0 fetched from AWS
2018-02-12 09:32:37,412 INFO Found subnet subnet-50be7f5f (10.0.0.0/24)
2018-02-12 09:32:37,412 INFO Polling for Salt Master public IP
2018-02-12 09:32:37,412 INFO Delegate 0 not instantiated
2018-02-12 09:32:42,418 INFO Delegate 0 not instantiated
2018-02-12 09:32:47,423 INFO Delegate 0 not instantiated
2018-02-12 09:32:52,428 INFO Delegate 0 not instantiated

[ibr123]

Hi,
Unfortunately, i had troubles with running the python scripts by getting errors, could you please guide me one how to create the requirements of AWS using the Console Management and a sample of the aws.yamil so i just fill it, then continue with the steps?
another question: why each delegate should be in its own subnet?

Thank you :)