Modularize IaC and adopt cloud-native tooling for k8s configuration

[derekmurawsky]

Adopt Modular IaC with Helmfile

Context and Problem Statement

Currently, everything is done via terraform, which is primarily an infrastructure tool. Though capable of managing Kubernetes resources, it is not the "best" tool for the job. Given the sprawling tools in the Infrastructure as Code (IaC) world, and the similar sprawl of tools in the Kubernetes configuration world, I propose the following changes to the repo structure.

Proposal

1. Move terraform code to a separate directory. Create an infrastructure/tofu directory and move all terraform files there. This allows us to add more IaC providers in the future without disrupting too much of the existing code.
2. Adopt a clean separation between infrastructure and k8s configurations. Right now there is no modularity, even in the TF code. This tight coupling will likely create problems as we scale. I propose treating they Infrastructure and K8s layers as distinct entities, with a clearly defined interface between them.
3. Adopt Helmfile and Create Helmfile directory structure. Following helmfile best practices, we will create the following directory structure:

- helmfile # A directory to contain all helmfile configurations
    - config # A directory to contain all helmfile configuration values
        - harbor # A directory to contain all harbor configuration values
            - values.yaml # A file containing default harbor configuration values
    - helmfile.yaml # The main helmfile configuration file

5. Utilize terraform to create configmaps - In order to cleanly pass required variables between the IaC layer and the Kubernetes/Helmfile layer, we will utilize terraform to create configmaps with required helmfile configuration values. These will be explicitly defined from the helmfile layer and treated as an API that the infrastructure layer must comply with. This will allow us to modularize the infrastructure layer more easily in the future.

Motivation

• Provide flexibility in IaC tool choice - Though Terraform is widely used, there are many other tools for IaC. Ideally, we would support many options in this space, like Pulumi, Crossplane, OpenTofu, CDK, Etc. By adopting a separation between infrastructure bootstrapping and application deployment, we can more easily swap out the tooling for each. This means we will be able to support more target environments in a cleaner way.
• Adopt cloud-native tools - Helmfile is a declarative spec for deploying helm charts, which is a more cloud-native way of deploying applications. It also allows for more modularity and reusability of helm charts.
• Allow for easier local development - Not everyone will want to stand up a complete AWS based kubernetes cluster. By separating the infrastructure and application deployment, we can more easily support local development environments using things like Rancher Desktop or k3s.

Considerations

• By separating the infrastructure and application deployment, we introduce more complexity into the pipeline. We will need to support multiple tools.
• We will need to develop a way to pass variables between terraform and helm configurations. There are many ways to do so, but we will have to settle on one.
  • Helmfile supports vals lookups from many sources, see github
  • Of the ones listed, we should likely use either File, or Kubernetes. File is simpler, but Kubernetes is more cloud-native and safer. Plus it supports secrets more easily.
• There are other configuration management tools for kubernetes. Helmfile is just one of them. Long term, we should consider supporting multiple tools in this space as well, just like in the IaC layer.