Open
Conversation
added 3 commits
May 15, 2025 15:17
- Clean up State Diagram - Define DRTPServer CRD - Define DRTPConnection CRD - Provide example manifests for Gateways and Routes - Add thoughts on how DRTP can support future End to End Connections.
thomasdaly5
changed the title
mksinghtx
reviewed
May 28, 2025
| Instead of engaging the technical support team: | ||
|
|
||
| * User Joe defines a DRTP service in the Portal/API. Joe provides a Service Name, Protocol (TCP or UDP), Local Hostname/IP Address, and Local Port Number. He is pointing at the data source he wants to expose to SelectStar. | ||
| * Datum Cloud responds with a $SERVICE_DETAIL and $SECRET, plus a link to download datumctl, and configuration instructions. |
Contributor
There was a problem hiding this comment.
datumctl serve can be downloaded as a container here that auto-starts and enrolls with datum-cloud using $SECRET. This method will secure better with certificates which have to be renewed. Automation is also a benefit.
command line invocation 'datumctl serve –service $SERVICE –secret $SECRET will not be required. User/Customers can be persuaded to do this I think.
|
|
||
| ## Future Opportunities | ||
|
|
||
| Scenario \#1: It may be the case that a Datum customer doesn’t want this service exposed to the Internet at all. In this scenario, it may be desirable to have `datumctl connect` act as a “forward proxy” on the client side, exposing the service internally, and then connecting upstream to the Datum Edge. In this case, no connections are allowed onto the Datum Gateway Service. Datum is now acting as a virtual “patch panel” for two instances of `datumctl serve` and `datumctl connect` with Datum Cloud in the middle as a virtual patch panel. |
Contributor
There was a problem hiding this comment.
Or use private addresses only for private connectivity. So establishing a private network using Galactic VPC between SelectStar and it's customers would make it possible to have private connectivity using native DRTP. The Route API in cloud providers can be used to get the private routes in both customer and SelectStar networks and stitched together via a global route table, path computation and path selection in Datum connecting the 2 networks. All services would be private then.
| On our roadmap, we have the concept of "App to App Connections." DRTP can be the basis for 25% of an App to App Connection. Looking forward: | ||
|
|
||
| - The first quarter of an App to App Connection is a `DRTPConnection` - this effectively gets a service onto an "inside-out" tunnel to the Datum infrastructure. | ||
| - The next quarter of an App to App Connection is a mechanism to securely initiate a connection from a client (in our PostgresSQL oriented example, say, Tableau Analytics) but without needing to have a priori knowledge of IPs / ports for an ACL. Imagine the command `datumctl connect` which: 1) Makes an outbound connection to Datum's Infratructure; 2) presents the remote PostgresSQL on `localhost:5432`. Magic wormhole! |
Contributor
There was a problem hiding this comment.
This is great ACLs and whitelisting are a pain !
|
|
||
| ### Future Thoughts on App to App Connections | ||
|
|
||
| On our roadmap, we have the concept of "App to App Connections." DRTP can be the basis for 25% of an App to App Connection. Looking forward: |
Contributor
There was a problem hiding this comment.
This looks good and having App level policies to control what Apps can connect to what apps will provide granular access that, visibility into these connections would be great !
mksinghtx
reviewed
Jun 2, 2025
| - The next quarter of an App to App Connection is a mechanism to securely initiate a connection from a client (in our PostgresSQL oriented example, say, Tableau Analytics) but without needing to have a priori knowledge of IPs / ports for an ACL. Imagine the command `datumctl connect` which: 1) Makes an outbound connection to Datum's Infratructure; 2) presents the remote PostgresSQL on `localhost:5432`. Magic wormhole! | ||
| - The next quarter of an App to App Connection is the "optimized virtual cross connect" - this requires a "software based letter of authorization" (e.g. authentication / policy), middle mile route optimization across the Datum backbone, and fast path failover. Network policy in Galactic VPC can be used to pick fast pass, low loss paths, or FEC enabled paths. | ||
| - The last quarter of an App to App Connection is the visibility of what is effectively a "socket across the Internet". Imagine being able to see the serving node telemetry (from the `datumctl serve` side) to the serving POP (the Rathole Server), to the middle mile network, to the client (the `datumctl connect` side)?!!! | ||
|
|
Contributor
There was a problem hiding this comment.
What I can also foresee is the first version of an AppCatalog being published to Datum subscribers based on scope of application access 1. Anyone over the web 2. Private connectivity over Galactic VPC 3. Restrict via policy eg to certain Geographic regions, Platform members only.
mksinghtx
reviewed
Jun 2, 2025
| * This causes a Gateway to be created, with EndpointSlices pointing to the 3x PodCIDRs where Rathole Server is running. The FDQN for the Gateway is returned to Joe for use. | ||
|
|
||
| * User Sam is now able to connect his client to his assigned “prism.global.datum-dns.net” FQDN on the port number he requested. In the case of SelectStar, he can provide them with the FDQN to reach the data source in question. | ||
|
|
Contributor
There was a problem hiding this comment.
Can we automate this so for example - Select Star is an organization on Datum Cloud and we publish the FQDN to subscriber orgs like Select star. The subscriber list is set by the owner of the Data source. It is essentially a service now available for Select star to access with appropriate privileges.
mksinghtx
reviewed
Jun 2, 2025
|
|
||
| * This causes `datumctl serve` to start, to connect to Datum Cloud, presenting its $SERVICE and $SECRET, to download a configuration file. It self configures with two or three connections to diverse Datum Cloud POPs where the DRTP server is running for upstream connectivity. At this point, the local firewall has been punched with an outbound connection. Simultaneously, `datumctl serve` also makes an outbound connection to the local hostname/IP/Port to connect to the service desired to expose. | ||
|
|
||
| * User Joe defines a Datum Gateway service, indicating that the DRTP service is the desired backend. Joe needs to provide a desired external port number, and MAY specify an IP whitelist for client connections. |
Contributor
There was a problem hiding this comment.
And/Or a list of orgs on Datum that are allowed to access the service.
mksinghtx
reviewed
Jun 2, 2025
|
|
||
| ## Future Opportunities | ||
|
|
||
| Scenario \#1: It may be the case that a Datum customer doesn’t want this service exposed to the Internet at all. In this scenario, it may be desirable to have `datumctl connect` act as a “forward proxy” on the client side, exposing the service internally, and then connecting upstream to the Datum Edge. In this case, no connections are allowed onto the Datum Gateway Service. Datum is now acting as a virtual “patch panel” for two instances of `datumctl serve` and `datumctl connect` with Datum Cloud in the middle as a virtual patch panel. |
Contributor
There was a problem hiding this comment.
Could private connectivity via Galactic VPC also be a way of not exposing this service to the Internet.
joshlreese
reviewed
Jun 13, 2025
| - `datumctl serve`: An extension to `datumctl` that provides the client side of the DTRP connection. | ||
| - DRTP connects to Datum Cloud on start, presenting its $SERVICE_DETAIL to unlock a bootstrapping API call to connect the Local Application Name/Type/Hostname/IP/Port and 2x nearby Datum Cloud POPs to connect to. | ||
| - Authentication and authorization is handled by existing workflows in `datumctl`. | ||
| - Uses `rathole-client` to make the connection to the $SERVICE_DETAIL and initiate the inside-to-outside reversed tunnel. |
Contributor
There was a problem hiding this comment.
Suggested change
| - Uses `rathole-client` to make the connection to the $SERVICE_DETAIL and initiate the inside-to-outside reversed tunnel. | |
| - Uses `rathole-client` (or similar) to make the connection to the $SERVICE_DETAIL and initiate the inside-to-outside reversed tunnel. |
enhancements/networking/research/datum-network-services/reverse-tunnel-proxy/README.md
Outdated
Show resolved
Hide resolved
| - Authentication and authorization is handled by existing workflows in `datumctl`. | ||
| - Uses `rathole-client` to make the connection to the $SERVICE_DETAIL and initiate the inside-to-outside reversed tunnel. | ||
|
|
||
| - rathole-server (or similar): Running in Datum Cloud POPs, is a daemon responsible for accepting inbound connections from `datumctl servce`, and turning them into reverse tunneled connections. Once established, we believe that these connections will be available on Datum PodCIDRs. |
Contributor
There was a problem hiding this comment.
Suggested change
| - rathole-server (or similar): Running in Datum Cloud POPs, is a daemon responsible for accepting inbound connections from `datumctl servce`, and turning them into reverse tunneled connections. Once established, we believe that these connections will be available on Datum PodCIDRs. | |
| - rathole-server (or similar): Running in Datum Cloud POPs, is a daemon responsible for accepting inbound connections from `datumctl serce`, and turning them into reverse tunneled connections. Once established, we believe that these connections will be available to leverage via additional Datum features, such as a Gateway backend. |
|
|
||
| - rathole-server (or similar): Running in Datum Cloud POPs, is a daemon responsible for accepting inbound connections from `datumctl servce`, and turning them into reverse tunneled connections. Once established, we believe that these connections will be available on Datum PodCIDRs. | ||
|
|
||
| - Datum Gateway: Our standard Datum Edge Reverse Proxy Service (Gateway API) configured with a TCPRoute/HTTPRoute and an EndpointSlice pointing at the PodCIDR of the appropriate rathole-servers. |
Contributor
There was a problem hiding this comment.
Suggested change
| - Datum Gateway: Our standard Datum Edge Reverse Proxy Service (Gateway API) configured with a TCPRoute/HTTPRoute and an EndpointSlice pointing at the PodCIDR of the appropriate rathole-servers. | |
| - Datum Gateway: Our standard Datum Edge Reverse Proxy Service (Gateway API) configured with a TCPRoute/HTTPRoute and an EndpointSlice pointing at the address of the appropriate tunnel termination servers. |
enhancements/networking/research/datum-network-services/reverse-tunnel-proxy/README.md
Outdated
Show resolved
Hide resolved
enhancements/networking/research/datum-network-services/reverse-tunnel-proxy/README.md
Outdated
Show resolved
Hide resolved
enhancements/networking/research/datum-network-services/reverse-tunnel-proxy/README.md
Show resolved
Hide resolved
|
|
||
| ### Registering a DRTPConnection (DatumCTL + Rathole Client) | ||
|
|
||
| By knowing our available DRTPServer nodes, by our K8s API "registry", we can now use `datumctl server` to create a connection. |
Contributor
There was a problem hiding this comment.
Suggested change
| By knowing our available DRTPServer nodes, by our K8s API "registry", we can now use `datumctl server` to create a connection. | |
| By knowing our available DRTPServer nodes, by our K8s API "registry", we can now use `datumctl serve` to create a connection. |
| - User identifies his/her local service to present via `datumctl serve`. | ||
| - `datumctl serve` connects to Datum control plane API to register the requested connection. The request includes $SERVICE_DETAIL (PostgresSQL/TCP/sqlserver.corp.com/5432), plus the client's detected WAN IP. | ||
| - The API service takes multiple actions: | ||
| - Hand back an array of DRTPServer objects to be used for `datumctl server` to configure Rathole Client. |
Contributor
There was a problem hiding this comment.
Suggested change
| - Hand back an array of DRTPServer objects to be used for `datumctl server` to configure Rathole Client. | |
| - Hand back an array of DRTPServer objects to be used for `datumctl serve` to configure Rathole Client. |
enhancements/networking/research/datum-network-services/reverse-tunnel-proxy/README.md
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The enhancement attached to this PR provide a 1st definition of DRTP - Datum Reverse Tunnel Proxy - a method for making "inside-out" connections to Datum infrastructure.