Materialization Triggering DBT Cloud: Hard coded to US. Want custom host/url to trigger github actions and DBT Core.

[usbrandon]

Greetings.

I love the idea of triggering DBT jobs to run at the end of a commit. I used to run DBT Cloud and am very familiar with the idea. At my current employer we are using DBT Core and Github Actions to trigger builds. While investigating Estuary Materialization source code with Will B's assistance, I noticed some obstacles to both using DBT Cloud, in regions outside the US, and in submitting a custom URL to suite my purpose of setting up a mechanism to trigger Github Actions that would cause DBT Core to run some content.

What I want:

• I want a way to supply Estuary with a custom hostname.

What I noticed:
https://github.com/estuary/connectors/blob/94728e6c03d9b23b95166b741e34e015190308cf/go/dbt/trigger.go#L65
Currently the code tries to be helpful and parse out the account ID used at line 65 to build the US specific URL for triggering job runs using their API.

To provide correct global coverage, that would only be complete if it also handled the URLs for other regions listed in DBT Cloud's documentation here: https://docs.getdbt.com/docs/cloud/about-cloud/access-regions-ip-addresses

I realize someone did not do that because they would need to deal with asking or allowing the user to specify their region. You'd need cloud accounts with DBT to test it too.

---

Back to what I really want. In order to create the least amount of work possible, I wanted to suggest the code changing a bit to allow me to put in a custom hostname, upon which all the other normal API paths and mechanics would try to work. I had in mind to create and release an opensource API proxy in the spirit of one that worked super well for me in AWS for taking OpenAI oriented tools, and letting them speak to other AI models https://github.com/aws-samples/bedrock-access-gateway

However, at the same time, an even better solution would be for Estuary to more directly support invoking Github Actions. Will B, kindly pointed out that Rivery.io does this https://rivery.io/blog/how-to-orchestrate-dbt-core-jobs/
I agree with that, but recognize that may be a bit more of an undertaking. I am willing to work with some rough edges of the custom host solution in the interim.

[dyaffe]

We will actually be adding a ton of functionality to this area soon which will enable us to work with Github actions. Our hope is to have that in Q1 but we can keep you updated as it's available.

The custom dbt URL bit is good information and we'll also look to add that.