Manage peer list from the Puppet master

[tux-o-matic]

The module is missing a key thing provided by central configuration management and which is available in two ways:

• Exported resources official doc
• PuppetDB lookup puppetdbquery module

With either one of these methods, the peer list could be fed on catalogue compilation by the Puppet master by collecting all etcd peers in the same cluster.

[cristifalcas]

I'm not sure I understand you.

If you are referring to initial_advertise_peer_urls, this can't be changed after the initial configuration (in puppet case after the first run). And only one peer will see the entire cluster (the last one that does a puppet run). And at least one will only see itself (the first one that executes a puppet run). Please correct me if I'm wrong.

If you are referring to the list that is received by the proxies, this can be updated and I could take a look on how to implement this.

[tux-o-matic]

@cristifalcas actually you might be able to address both scenarios.

1 - Initial setup

• Prevent Puppet to apply the configuration until at least two etcd nodes are found with puppetdbquery on catalog compilation. With PuppetDB, since a $fqdn array would be returned this would require to separate the protocol scheme and port as class parameter to build the URLs for 'initial_cluster'.
• If this concept works, the result is that 'initial_cluster' no longer needs to be specified.
• If you also change some of the default values (for example of 'initial_advertise_peer_urls') from using 'localhost' but instead $fqdn then you could strip a few of the value needed for a basic cluster setup. The module would be way more "convention over configuration".
• In pseudo code you could do something like:

class etcd (
 $advertise_port = 2380,
 $advertise_protocol = 'http',
 $initial_peer_requirement = 0
) {
 if $initial_peer_requirement == 0 {
 #Good old manual configuration
 } else {
 $configured_peers = query_nodes("Class[etcd]")
 if size($configured_peers) < $initial_peer_requirement {
  notice("Not enough etcd nodes found")
 } else {
  #concat $configured_peers with ${advertise_protocol} and ${advertise_port}
 }
 }
}

2 - Adding peer to an existing cluster

• Regardless of initial cluster setup, use information avialable to the Puppet master about existing etcd nodes configured to allow a new etcd peer to join a cluster.

[KlavsKlavsen]

I do something like this --for elasticsearch clusters:

    $pdbquery = ["from", "resources", ["and", ["=", "type", "Class"], ["=", "title", "Profile::Elasticsearch"]]]
    $es_data_nodes = delete_undef_values(puppetdb_query($pdbquery).map |$res| {
      if ( $res['parameters']['data_node'] == true and $res['parameters']['cluster_name'] == $cluster_name ) {
        $res['certname'] #relies on certname being dns resolvable !
      }
    })

This way i only use nodes who have the same clustername

[cristifalcas]

But at the very first run one of the nodes will run first and be the first to query who has that class and it will only see itself. And it will bootstrap the cluster with only this info.

On subsequent runs, the other nodes will be added to the list also, but for etcd is too late.

For elasticsearch you can modify the masters list later, but not for etcd.

If I am wrong, please let me know

[KlavsKlavsen]

I just wrap that in a

if ( size($etcd_nodes) >= 3 ) { 
  class { 'etcd': ..
} else {
  notify { "Waiting until we have 3 nodes or more joined - to setup cluster.": }
}

[cristifalcas]

It could be done with a new parameter cluster_size. And don't do anything until we have exactly that number of instances with the etcd class.

Having said that, I'll appreciate any PR for this, but I will not have time to do it myself.