Self-hosting with Kubernetes/Docker

[davegaeddert]

Currently PullApprove Enterprise (the self-hosted version) is deployed via Terraform onto AWS Lambda and a handful of other AWS services.

In the past week there have been several questions about whether a Kubernetes option is available and I want to be open to that idea!

Here are some of the reasons, from what I can tell:

• A lot of orgs that self-host these kinds of things are using Kubernetes internally now
• GitLab also bundles some Kubernetes deployment tools/features
• The AWS/Terraform setup has grown in complexity over the years (I'm not sure how much simpler a Kubernetes option will end up being, but it may be more familiar and flexible at this point if nothing else)
• I've never totally ironed out the best workflow for distributing the Terraform files -- some people customize them a lot, and some don't at all. Some run forks/clones of the repo and some use it as a module. I'm getting the sense that people would prefer a less opinionated setup here (i.e. a well-documented Docker image they can run) and then they can make their own decisions around HTTP routing, secrets management, etc. that fit the way they're doing it with other tools

I think getting PullApprove onto Kubernetes is a totally viable idea. I doubt that it will make sense for it to run as a single container though. Mostly due to webhook timeout requirements -- GitHub uses a 10 second timeout and depending on size of a PR etc. it can take longer than that for PullApprove to run. So, my initial guess at what we'd end up needing to run would be:

• a webhook container
• a Redis container (for queueing, and API request caching since we have it instead of the filesystem cache used today)
• a worker container to process the queue
• an AWS S3 bucket for reports storage (same as today)
  • an AWS IAM user for uploading / generating signed URLs
• an AWS S3 bucket for the public report UI and docs (same as today)

What would people expect for an install / upgrade / deployment process? Configure it entirely on your own if given a PullApprove Docker container (or two)? A Helm chart? Anything with Terraform (some minimal S3 (or S3 compatible) services would still be required, I think)?

On the topic of hosting options and Docker containers, I should also add that I've considered whether pullapprove could run as a GitHub Action / GitLab Pipeline / etc. The biggest downside to this would be that pullapprove responds to a lot of events that aren't triggered by those systems, so it would probably fall back to a semi-run-on-demand kind of situation because the status could easily be outdated (not great from a compliance perspective). It would also trigger a lot of CI runs if used like it is today. If this sounds like a potential direction, let me know...

Anyway, I just wanted to start a public discussion in case people want to chime in here. If we shift to Kubernetes for self-hosting PullApprove, my preference would be to drop the Lambda support for sake of maintainability. So I really would want to make sure this direction is fully thought through before taking it on, and a lot of that needs to be influenced by the people using it! You can email me if you prefer to discuss details in private (I can post relevant points back here), but otherwise we would probably all benefit from this being a shared conversation. Thanks!

[blockjon]

It's not that it needs to run on kubernetes per se, it needs to be shipped via Docker images. Your customers can deal with rigging a way to run the Docker images (in our case, a kubernetes pod).

If you can just ship it as Docker, it can run the same way everywhere on any cloud on Kubernetes, ECS, Nomad.. whatever.

Just yesterday, we spent hours trying to fight with terraform to launch more lambadas and it just makes very little sense from my vantage point to continue to do things that way. Lambda is the one thing all Amazon customers have but that nobody wants to use because the tooling isn't very good.

[davegaeddert]

Thanks @blockjon. So even if it ends up requiring Redis, multiple PullApprove containers, etc. you're saying that the only things you'd want me to provide are the PullApprove docker images and some documentation, right? I pulled Kubernetes back in because everyone who has mentioned Docker has also called out Kubernetes by name, but I would be totally fine with providing just the Docker part and being more agnostic on how it's deployed. I think some people could be looking for a more pre-packaged process but maybe good documentation/pointers is better at this point than trying to provide a Helm chart or anything of that nature.

Anybody else want to leave a little 👍 (just give me Docker) or 👎 (I want more than that) reaction at least?! Any comments welcome.

[blockjon]

Yes - you as the PA maintainer don't need to "provide a redis container" - you just tell us which one on DockerHub we can run at the version tag that is supported. In our case, we would probably use AWS Elasticache over a redis image.

[davegaeddert]

In our case, we would probably use AWS Elasticache over a redis image.

Got it -- good to know!