Make Harness work with load balancer

[pferrel]

Synchronize Harness server means that all must know when one has updated the metastore. I think this means minimally:

• one Harness server needs to know all others. maybe by one being the temp master who maintains the db of all others? Is there a simpler way? Like using Etcd or Consul?
• load balancing handled by Kubernetes or regular load balancer but sync handled by the Harness cluster
• Harness could have a sync REST API, which tells Harness to re-sync with the metastore
• sync sent to all harness servers when:
  • any server executes harness add, harness update, harness delete, or ???

Not sure of what else should be done. Any cluster of Harness needs to keep track if all others so probably a master voting structure like ES, where one keeps the DB of all other server, and if the master is not available another is voted to take over.

This seems pretty complicated so is there an easier solution?

Are there signaling DBs that will notify listeners when something changes?

[pferrel]

we have a branch where this is working by using Etcd for pubsub of engine changes. This allows multiple Harness instances to coordinate the only REST operation that changes Harness state -- namely harness add, harness update, harness delete.

This is being moved to the Harness v 1.0 release, coming after the 0.6.0 release.