feat: add GPU-enabled self-hosted runners for vLLM recording

[cdoern]

Summary

Add support for recording vLLM integration tests on GPU-enabled EC2 instances with the gpt-oss:20b model (20B parameters). This enables testing larger models that don't fit on standard CPU runners while maintaining cost efficiency.

Key Features:

• OIDC authentication - No long-lived AWS credentials stored in GitHub
• Multi-region/AZ fallback - High availability across 9 availability zones (us-east-2 and us-east-1)
• Security hardened - Test jobs run with zero permissions to prevent credential theft
• Always-cleanup - EC2 instances terminated even on failure or cancellation
• Cost efficient - ~$0.43 per run on g6.2xlarge

Components:

• .github/workflows/record-vllm-gpu-tests.yml - Main workflow with manual trigger
• .github/actions/launch-gpu-runner/ - EC2 instance launcher (wraps machulav/ec2-github-runner)
• .github/actions/setup-vllm-gpu/ - vLLM GPU installation and server setup with AWQ quantization
• Test configurations for vllm-gpu-gpt-oss setup

AWS Setup Required

This PR includes all code but requires AWS infrastructure setup before it can be used:

1. OIDC Provider + IAM Role - See AWS_SETUP_GUIDE.md Step 1
2. VPC and Networking - Subnets, security groups in 2 regions
3. GPU-Enabled AMI - CUDA 12.4, Docker, NVIDIA Container Toolkit
4. GitHub Secrets/Variables - Configure AWS_ROLE_ARN, subnet IDs, AMI IDs, etc.

See IMPLEMENTATION_STATUS.md for detailed checklist.

Test Plan

Once AWS infrastructure is set up:

• Trigger workflow via Actions tab > vLLM GPU Recording > Run workflow
• Verify EC2 instance launches in us-east-2
• Verify GPU detected and vLLM server starts with gpt-oss:20b
• Verify tests run successfully in record mode
• Verify recordings uploaded as artifacts
• Verify EC2 instance terminated automatically
• Run 5-10 times to validate reliability

Documentation

• User Guide: docs/gpu-runners.md - How to trigger workflows, troubleshooting, cost estimates
• AWS Setup: AWS_SETUP_GUIDE.md - Step-by-step infrastructure setup with OIDC
• Implementation Plan: IMPLEMENTATION_PLAN.md - Roadmap and future optimizations
• Status Tracker: IMPLEMENTATION_STATUS.md - Current status and next steps

References

• Design based on instructlab/instructlab and opendatahub-io/data-processing GPU runner patterns
• Uses machulav/ec2-github-runner for EC2 instance management

[cdoern]

( I will remove the annoying MD files once this is ready )

[iamemilio]

It might be a little easier to keep a mapping of instance types and fallbacks to each supported model. That way users don't need to know or configure this when setting jobs up. That way we can also take into consideration the different resource requirements of the models. For example qwen 3.5 0.8B doesn't need an L4, it can probably run on a g4dn.xlarge and do reasonably well. To make your life easier for v1 of this pr, its probably easier to support just one model for now.

[iamemilio]

You may want to move this into a script, so that its testable. I don't think its doing what we think it is. I'm pretty sure this only ever uses us-east-2 due to the iterator being hardcoded to 0:

IFS='|' read -r REGION AZ SUBNET AMI SG <<< "${CONFIGS[0]}"

[courtneypacheco]

Ah, this version of the GH action actually allows for setting different availability zones!

See here: https://github.com/opendatahub-io/data-processing/blob/main/.github/workflows/execute-all-notebooks.yml#L49

Let's use this approach so that we can try different availability zones in case the first one lacks availability. Then let me know which AWS regions you need the AMI ID to exist in.