enhance flow for custom machineconfigs for specific machinesets

[cgwalters]

The CI team is trying to use new AWS m5d.xlarge instances which have two NVMe disks attached. We crafted a custom RAID partition machineconfig to enable that.

We added this as part of the main worker pool - the MCD will fail to roll out the partitioning on the existing workers, but that's fine because the plan was to "roll" the worker pool. Basically get the new MC in the pool, have new workers come online with that config, then scale down the old workers.

However, there are a few issues here.

First, this whole thing would obviously be a lot better if we had machineset-specific machineconfigs. That would solve a bunch of races and be much more elegant.

What we're seeing right now is that one new m5d node went OutOfDisk=true because it was booted with just a 16G root volume from the old config. That unschedulable node then blocks rollout of further changes.

I think we can unstick ourselves here by deleting that node and getting the MCO to roll out the new config.

[cgwalters]

And the real problem here is that the MCS serves:

currConf := mp.Status.Configuration.Name

So new nodes will only get the last config that successfully rolled out, so this attempt deadlocks.
We really need machineset-specific MCs.

[cgwalters]

Another idea: some sort of "force MCS to serve mp.Spec.Configuration.Name" config option?

[cgwalters]

First, this whole thing would obviously be a lot better if we had machineset-specific machineconfigs.

This though gets into the "node identity" problem - see also #784
Which could be e.g. a secret value passed through user data and then provided to the MCS, which can link that to a machine object, and from there a machineset.

[cgwalters]

If someone wants to change the machine type of their workers today I think it'd require a custom worker-new pool and also creating a custom user-data that does s/worker/worker-new/ that's referenced from the new machineset.

Alternatively, scaling down to 0 workers then back up should work, but obviously that's a lot more disruptive.

[cgwalters]

The other alternative would be having something like a spec.machineSet in a pool - then the MCO could understand how to say "okay, the new config has partitioning changes, so let me destroy the machine objects and provide new configs to newly provisioned nodes". That'd be the most elegant but would require tightest integration between MCO and machineAPI.

[cgwalters]

An entirely different idea is for something in the stack to have a high level tunable knob like:

storage:
  instanceAttached: /var

This would be a MachineConfig fragment.

This logic could potentially live in the MCD; rather than having a machineConfig object describe a specific number of disks, the MCD would dynamically look at the node and say "oh this instance has 2" and set things up accordingly.

The tricky part with this is that simply waiting for "all disks" gets racy. It'd really be best done with knowledge of how many disks to expect. We may need some integration with the machineAPI so we know how many disks to expect per instance type (per configuration) - it feels dangerous to just unilaterally take over block devices we see, though maybe there are provider-specific ways to know they're ephemeral disks.

[michaelgugino]

This machineconfig provided in first comment: openshift/release#8102
And custom worker instructions: https://github.com/openshift/machine-config-operator/blob/master/docs/custom-pools.md

Work together well. All that you need to do is create a custom user-data secret. This can be done easily.

First, find an existing machineset you want to modify, and download the user-data secret.

./oc get secrets -n openshift-machine-api worker-user-data -oyaml > worker-user-data.out.yaml

Modify scrape the secret contents into a new file, base64 decode those contents (you'll get some json), edit the URL near the beginning of the JSON to specify the new mcp name you created. Base64 re-encode that modified json (make sure to use -w0 option so you don't get line wrap), paste that into the user-data yaml file, update the name to something sensible and oc apply.

Update your preferred machineset to utilize the m5d.4xlarge instance type and to utilize the new user-data secret you created. That's it.

[cgwalters]

Modify scrape the secret contents into a new file, base64 decode those contents (you'll get some json), edit the URL near the beginning of the JSON to specify the new mcp name you created.

This part is pretty user hostile I think. But doing better requires much tighter MCO/machineAPI integration; something like having the machineAPI ask the MCO "please give me the user data secret for pool X" or so.

[michaelgugino]

Modify scrape the secret contents into a new file, base64 decode those contents (you'll get some json), edit the URL near the beginning of the JSON to specify the new mcp name you created.

This part is pretty user hostile I think. But doing better requires much tighter MCO/machineAPI integration; something like having the machineAPI ask the MCO "please give me the user data secret for pool X" or so.

@cgwalters

We more or less are doing that by specifying a user-data secret.

The MCO could stamp out user-data for each pool, and users could just update/create a machineset that points to that secret.

[cgwalters]

I think in general a good pattern would be using /var for an ephemeral drive - but to make that work right we need to move e.g. the pull secret into /etc. Same reason we want to do that for #1190