Create a persistent dashboard to see all ray clusters and history

[jhasm]

The Ray dashboard service runs on the ray head node, which goes away after the ray cluster is deleted. This makes the ray dashboard of little use when you are running ephemeral or short lived ray clusters. Long running ray clusters have reliability issues. So the ray clusters get an impression of limited transparency and observability.

If we can move the dashboard service outside the ray cluster, we can have a global dashboard that can show the jobs across all the ray clusters, even after the cluster is deleted. This needs to be backed by a dedicated data store, which gets data from the ray cluster's dashboard service, while the cluster is still running. This will allow us to preserve the history of all the rich data that we see on the ray dashboards, independent of how long the ray cluster lives.

In the nutshell, we need a way to export the ray dashboard data to an external data store, while the cluster is running. And we need an external dashboard service that shows the exported data. The external dashboard can have the same GUI as the ray dashboard service, to keep it consistent.

[jhasm]

I am happy to collaborate on this feature, since this is important for overall ray usage at my company.

[pdames]

Agreed that this would be a very valuable addition @jhasm, and would love to work together on this. I've had similar thoughts, as this also aligns with DeltaCAT's goal of providing a durable data store for Ray (e.g., one way to model the dashboard's dedicated data store could be as a stream of DeltaCAT Deltas read from the Ray Dashboard Service - this would ideally allow us to both rehydrate historic dashboard metrics, analyze logs/metrics via Daft/Pandas/Polars/etc., and synchronize to Iceberg to query via Spark/Trino/etc).

[pdames]

Hi @jhasm - this may be more practical to approach now that we've merged #570, which stabilizes the DeltaCAT 2.0 catalog APIs and provides usage examples.