Support for atomic multi-stage app deployments

[christianparpart]

WIP

GOALS:

Add support for atomic deploys while retaining support for rolling deploys.

DEFINITION: "Rolling Deploy"

Upon redeploying a application cluster, the new application instances (tasks) are spawned within the existing cluster. Once a new task becomes healthy an old task is being terminated, leaving the cluster with two versions during the deployment phase. Only when all tasks have been fully replaced with the new deployed version, the app cluster is not mixing up responses anymore.

DEFINITION: "Atomic Deploy"

An atomic deploy for a load balanced cluster is where the newly deployed tasks are activated in the load balancer (/service discovery) once all tasks became haealthy. The LB will then replace all old task's endpoints with the new endpoints while current sessions are still allowed to complete. Only after that the old deployment is subject to be terminated.

Motivation

In current deployments, we see somm 5xx HTTP response status codes due to the fact that we're having rolling deployments, and we hope to reduce them with the atomic switch from an old to the new app cluster version.

Implementation

By default, we consider each application to be a blue app. We can additionally link a green app to the blue app with an application label.

• blue: /production/frontend-webapp with label mmsd_atomic_deploy_suffix set to -green.
• green: /production/frontend-webapp-green

Properties of Linked Applications

• They must be the same application that only differ in functionality (such as feature or bug fixes)
• The service discovery ports from the blue application will be used by mmsd to expose the active deployment, whether it is the blue or the green app.
• The application that gets deployed the last is the active application.
• mmsd will not update the apps load balancer configuration during the deployment of a linked application,

Notes

Now, when an application gets redeployed, mmsd is getting notified via the event bus by the time the deployment starts.

• mmsd will update its local state and fetch all applications metadata, including the new mmsd_atomic_deploy_suffix and build an internal map of blue/green apps.
• When the deployment of a linked application has completed with deployment_success, mmsd will reconfigure the load balancer to point to the latest deployment (blue or green).

Editorial Notes

• the name of the label for linking two apps into one is not fixed. we can choose some better name, if any (OTOH: doesn't matter that much)
• this ticket is work-in-progress and subject to drastic changes.

[markauskas]

Question 1: are the service ports for blue and green identical? If not, what port will a client application use to connect to an app? e.g. if blue port is 10101 and green port is 10102, how can a service, which depends on the blue/green service, connect to it?

Question 2: How do we rollback? If I deploy green and blue is still alive and I notice that green is buggy, how do I rollback?

[markauskas]

Possible alternative to blue/green: we could deploy blue/green services as /marathon/app/id/VERSION_ID, e.g. /production/frontend/webapp/version-23. Then we would just make sure that mmsd always points to the desired deployment.

This doesn't solve any problem, it's just an alternative way of structuring the hierarchy of marathon app IDs.

[christianparpart]

@markauskas IMHO, the question then would be of what service port to use. ideally I'd like to preserve the idea of letting Marathon maintain the service discovery ports, so mmsd doesn't introduce state.

[markauskas]

That is exactly my question. We either need to use one service port for all of these deployments and mmsd would only use the current deployment to configure haproxies, of we would need a global service port that would map to the service port of each deployment, but then we obviously would need to maintain this mapping somehow.

[christianparpart]

Answer to Q1: Marathon is just proposing a (managed list) service discovery port. mmsd uses that information to expose this port on the load balancer. I thought (so far) it may be best to use the main app (not the linked app) mandates the service discovery ports.
With this approach it may be easier to switch back from blue/green to rolling deployments. like a checkbox.

Answer to Q2: Rollback may be implemented in the (your) deploy app with a flip button (or rollback button) that flips between the 2 live versions. This flip action could then send a POST to mmsd to replace the backends with the other deployments backends of your app X. (open for discussion).

Marathon's rollback equals redeploy within the same app. So that would take just too long compared to the proposal to answer Q2.

[markauskas]

Would the deploy tool need to send a request to every mmsd instance running on each hardware node?

[markauskas]

It seems to me that sending a request to mmsd would mean that mmsd needs to keep it's own state somehow. And it would not be possible to infer the desired state by just reading the currently configured apps in marathon. So if mmsd crashes, it needs to retrieve the state from marathon AND from somewhere else.