[DISCUSS] remote-exec support

[jaketf]

Description

Describe the purpose of this pull request.

This PR contains a rough skeleton of what it would look like to support remote execution.
Before I go further I want to open this up for some review feedback to make sure this is a direction we want to pursue.

the approach was to define a new remote-docker-compose.yaml and a remote-exec package that basically re-uses most of the airflow packages docker utilities for astro dev start|stop|kill to provide a similar experience for running remote agents locally.

Purpose

Currently, the purpose of this PR is to make a demo / POC for discussion and to assess the value of further investing in this feature.

Going forward with this would increase our maintenance surface area and here is a decent amount more work required here just to get to MVP, and there are some clear next steps to make this really nice for users described in #1880.

I personally think this would make it much smoother to register astro-agents from non-kubernetes environments.

Status:

this almost works but needs to do something to side-load the astro-agent stuff in as right now it just reads the runtime image from the remote deployment in core API. @ianbuss @neel-astro curious what you would recommend for this. I imagine we could develop a mechanism to dynamically pull the latest agent release and mount it into the proper path.

TODOs

• right now if things fail it will print error messages related that refer to the astro dev * commands which is midleading. We should refactor to properly pull the docker / compose logic out into a common package and have the airflow and remote-exec handle these error messages.

   Error: There might be a problem with your project starting up. The api-server health check timed out after 60ns but your project will continue trying to start. Run 'astro dev logs --api-server | --scheduler' for details.
  

Try again or use the --wait flag to increase the time out```

• add unit tests
• integration / e2d test with a remote execution deployment.

🎟 Issue(s)

Related #1880

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

• Rebased from the main (or release if patching) branch (before testing)
• Ran make test before taking out of draft
• Ran make lint before taking out of draft
• Added/updated applicable tests
• Tested against Astro-API (if necessary).
• Tested against Houston-API and Astronomer (if necessary).
• Communicated to/tagged owners of respective clients potentially impacted by these changes.
• Updated any related documentation