Add SSHLauncher for multi-node execution

[sfc-gh-truwase]

Add SSHLauncher for multi-node execution to complement existing the Slurm and MAST launchers. SSHLauncher is applicable when the nodes are already preallocated and accessible via passwordless SSH. This is related to earlier discussions in #587 #611

SSHLauncher can be enabled by specifying ssh as launcher in the provisioner section of yaml as follows:

provisioner:
  launcher: ssh
  ssh_hostfile: /job/hostfile
  colocate_ref_and_trainer: true

/job/hostfile is assumed to contain a list of hostnames (or SSH aliases), which are machines accessible via passwordless SSH, and slot counts, which specify the number of GPUs available on the system

 worker-1 slots=8
 worker-2 slots=8

This PR currently includes an option to colocate trainer and ref_model.

Test plan

[meta-cla]

Hi @sfc-gh-truwase!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[sfc-gh-truwase]

@daniellepintz @allenwang28 will appreciate your feedback.

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[allenwang28]

Thanks for opening this @sfc-gh-truwase!

Can you share more information about your preferred cluster setup?

I see right now that we're grabbing from a host names text file, not sure if that's something you're creating yourself or if that's something your scheduler creates automatically for you. Some schedulers like SLURM may e.g. give you SLURM_JOB_NODELIST or something similar

[sfc-gh-truwase]

@allenwang28 yes, in our cluster setup the scheduler automatically creates the text file of allocated host names. Do you have suggestions for generalization?

Regarding, SLURM, I assume that existing SlurmLauncher would suffice. Does that align with your thinking?

[allenwang28]

ok, understood. For context, Monarch has recently created a "Job" API which defines how Monarch acquires HostMeshes from underlying schedulers. See SlurmJob for instance.

There's also an implementation for SSHJob.

I'm planning on migrating Forge over to the Jobs API, and away from the launcher specific APIs that we have today. I'm wondering if this would work for your cluster? We could probably test a much simpler workload than Forge for this.

[sfc-gh-truwase]

@allenwang28 yes, we used SSHJob in our PR. Can you advise if the usage aligns with your plans?

[daniellepintz]

Hi @sfc-gh-truwase, thanks for your work on this launcher! As Allen mentioned we are working on a refactor to all of our launchers at once, which involves substantial changes to provisioner.py and SlurmLauncher, etc. Would you mind waiting until that refactor is done, and then rebasing your SSHLauncher changes on top? Would greatly appreciate it! That refactor should be done later this week

[daniellepintz]

Hi @sfc-gh-truwase thanks for your patience! I have merged #700, would you like to take a look and see if the API works for you?

[sfc-gh-truwase]

@daniellepintz @allenwang28 I have aligned the PR to the new API. Will appreciate your review. Thanks!