Added example training scripts for localsgd, DiLoCo, Live Checkpoint Recovery, and proactive failure detection with DDP (#198)

[WarrenZhu050413]

TorchFT Examples

This PR adds a comprehensive set of examples demonstrating various fault tolerance features and distributed training approaches in TorchFT. Each example includes a per example README that provide step-by-step instructions, and sample outputs to help users understand and incorporate these features into their training script. All the examples build on top of train_ddp.py.

The PR came from my own experience understanding the different features of torchFT. I found it hard to start running other features outside of the given train_ddp.py at the beginning, which made it more difficult for me to have a sense of the various features offered by torchFT.

@d4l3k provided useful feedback in how to structure the examples.

Examples Included:

1. DDP with Proactive Failure Recovery (examples/ddp_proactive)

   • Demonstrates how to enable proactive detection and response to worker failures
   • Includes detailed explanation of recovery mechanism with annotated logs
   • Shows significant reduction in recovery time compared to timeout-based approaches

2. DiLoCo (Distributed Local Convergence) (examples/diloco)

   • Implements DiLoCo training methodology
   • Shows how to configure and optimize local convergence parameters
   • Documents performance characteristics and tradeoffs

3. LocalSGD (examples/localsgd)

   • Demonstrates LocalSGD with periodic synchronization strategy
   • Provides guidance on setting appropriate synchronization frequency
   • Includes performance comparison considerations

4. Live Checkpoint Recovery (examples/live_checkpoint_recovery)

   • Shows how to implement checkpoint-based recovery for fault tolerance
   • Documents the checkpoint storage and retrieval process
   • Includes recovery time analysis and optimization tips

[H-Huang]

Wow! This is really awesome, thank you for your contributions. I will take a look more closely soon.

One thing I think would be a good idea is to run these examples as part of CI as well so we know we aren't regressing anything. We might have to create a separate CI workflow for this.

[WarrenZhu050413]

@d4l3k also commented on this. Working on the CI rn!

[WarrenZhu050413]

Added CI.

This is the output when I run examples/test_examples.py locally. The tests are currently CPU only, using torchx.py's default.

For the .yaml file, the environment is set up identically to torchft/.github/workflows/unittest.yaml.

> pytest examples/test_examples.py
======================================== test session starts =========================================
platform linux -- Python 3.11.11, pytest-8.3.5, pluggy-1.5.0
rootdir: /srv/apps/torchft
configfile: pyproject.toml
plugins: typeguard-2.13.3, timeout-2.3.1, anyio-4.9.0
timeout: 60.0s
timeout method: thread
timeout func_only: False
collected 12 items                                                                                   

examples/test_examples.py ............                                                         [100%]

=================================== 12 passed in 99.86s (0:01:39) ====================================