RFC: Terraform Module Versioning vs Project Versioning

[novekm]

Is this related to an existing feature request or issue?

No response

What part of the Cloud Game Development Toolkit does this RFC relate to?

Modules

Summary

This RFC is to gain feedback on the current function of the versions of the toolkit, and to discuss the potential to add versioning to the Terraform modules that are within the toolkit.

Use case

The reasoning behind this suggestion is because currently there is no way to reference a specific version of any of the Terraform modules. We have the version of the toolkit, but there is no link between this and n number of changes that could take place to any of the modules.

Proposal

Currently the toolkit is on v1.1.3-alpha however, what does this mean for an end user? This project mentions "The Cloud Game Development Toolkit (a.k.a. CGD Toolkit) is a collection of templates and configurations for deploying game development infrastructure and tools on AWS." At its core, these templates are largely Terraform modules, since the goal of the project is to make it easier to deploy infrastructure. There are things like Packer templates, and Ansible playbooks as well. These can be consumed standalone, however it's my understanding that the typical expected usage is to use these in tandem with a Terraform module that would be provisioning the actual infrastructure in AWS.

However, currently there is no way for a user of the toolkit to reference a specific version of any of the Terraform modules. For example, assuming there is a request to add SES email support for Perforce Helix Swarm. If and when this is added to the Perforce module, how can a user effectively use this new enhancement that was added to the module?

In a standard Terraform module, this would be achieved by referencing the specific version of the module that adds the functionality. For those who do not desire to have this functionality, they could continue to use their current version of the module. The same applies for Terraform providers, such as the AWS Provider for Terraform. At the moment, when a new release of the toolkit is cut (e.g. v1.1.4-alpha), this can provide any number of changes to these Terraform modules (as well as the Packer templates, Ansible playbooks, etc.) however this is potentially destructive to the Terraform modules. This version of the toolkit has no relation to the version of any of the Terraform modules in a way that can be referenced by a Terraform user.

Recommendations

In addition to the version releases for the toolkit itself, I propose that we start to version the Terraform modules that are being developed in the toolkit. This will allow us to more effectively track changes that are made to these modules, align feature requests and the related completed PRs to the versions, and allow end users to reference specific versions of the modules as desired.

I propose that we standardize using the Terraform Registry and individual git repositories to host all of these modules for the following reasons:

1. Easier visibility, lifecycle management, and contributions

The Terraform Registry is the standard location Terraform developers go to to find modules that they can use. By using the registry, it is easy for developers to use specific versions of the module and the remote source of the registry itself. For example:

module "iam-identity-center" {
  source  = "aws-ia/iam-identity-center/aws"
  version = "1.0.2"
}

This also means that potential users who may have a use case for something in the toolkit but are not a direct game studio can also make use of the modules we are actively developing. This can increase the number of community contributions we receive and ultimately lead to advancements in the modules with less of a reliance on internal contributors.

For a data point, the AWS IAM Identity Center Terraform Module has ~50k downloads, and 2 of the most major changes were submitted by external contributors.

2. Enhanced usage tracking and alignment with project roadmap

Another benefit of the Terraform Registry is the usage metrics they provide. This is a clear data point that can be used to determine the most in demand modules, and prioritize adding new functionality based on this.

For example, take two modules released in the AWS-IA GitHub Org:

• AWS IAM Identity Center (~50k downloads
• AWS Amplify App (~1k downloads)

The AWS IAM Identity Center module was released almost a full calendar year after the AWS Amplify App module, however it has almost 50x more downloads in 50% of the time. For prioritization, it is clear which would take preference if both had a feature request.

Example Usage

module "perforce" {
  source  = "aws-games/perforce/aws"
  version = "1.0.0"
}

module "jenkins" {
  source  = "aws-games/jenkins/aws"
  version = "1.0.0"
}

module "teamcity" {
  source  = "aws-games/teamcity/aws"
  version = "1.0.0"
}

Out of scope

Anything that is not a Terraform module.

Potential challenges

A potential challenge is the method used for Terraform module versioning. Typically a Terraform module is contained to a single git repository. This can then be used as the source when storing the module in the Terraform Registry. To my understanding, one of the main reasons a single Terraform module should be used within a single git repo is because of git tags. These are what are used to tag commits and release new versions of the module. If continuing down the current path of keeping all Terraform modules within the core Cloud Game Development Toolkit repo, I do not think we will be able to add git tags to the modules that are in the subdirectory. And even if we were able to, a single Terraform module hosted on the Terraform Registry must be contained to a single git repository. As such, I propose that we break out the modules currently in the toolkit into individual standalone git repositories.

While this has the con of increasing the number of GitHub repos we deal with, I believe the pros outweigh the cons, especially for the end users. The toolkit can think reference these individual Git repositories for ease of visibility. For an example, see [AWS Labs] MCP(https://github.com/awslabs/mcp?tab=readme-ov-file).

Dependencies and Integrations

No response

Alternative solutions

[novekm]

Relevant discussion from HashiCorp discuss: https://discuss.hashicorp.com/t/versioning-modules-in-a-monorepo/17650/2

Users of the CGD Toolkit currently fork or clone the repository, and either deploy directly or copy and paste module source into their own IaC projects. Changes to module functionality have been minimal thus far, so version bumps have been inconsequential.

Should a user wish to avoid a particular change or implementation of a module they can cherry pick in commits as they see fit. Some of our end users have already expressed that this is the primary way they are consuming the Toolkit - as a managed Git submodule.

As such, it is possible to consume distinct versions of our modules - its just that we use git to do this as opposed to the TF registry. As for using the Terraform registry, I'm not sure how I feel about the complexity of splitting each module into a separate repository. If this is something we actually want to undertake there is significant scoping required, and it takes time and effort away from delivering valuable updates to the project. I'm game for it, but I have a couple of questions we should answer first:

1. How do we manage Packer and Ansible playbooks related to a workload? Does the Perforce Helix Core Packer template live in the Terraform module for Perforce Helix Core? Does the Ansible playbook for registering Horde agents exist inside the Horde module?
2. Does the Cloud Game Development Toolkit continue to exist as a collection of examples connecting these modules together in complex architectures? Do we publish it as a module as well (quickstart idea)?
3. Do the examples in each module only pertain to that technology? In other words, where do we store a sample configuration that deploys Helix Core, Helix Swarm and the Helix Authentication Service? Maybe the solution is all three of these modules exist together and are released and versioned together?

I'm also a bit unclear on the MCP example. This project is a monorepo that contains source for multiple MCP servers. The issue actually relies with Terraform registry constraints on versioning to a single Github repo - plenty of package management tools actually support the ability to release distinct versions within a single repository.

[novekm]

1. How do we manage Packer and Ansible playbooks related to a workload? Does the Perforce Helix Core Packer template live in the Terraform module for Perforce Helix Core? Does the Ansible playbook for registering Horde agents exist inside the Horde module?

If a specific module directly depends on a Packer template and/or Ansible playbook (e.g. this is a pre-requisite to be able to successfully deploy resources using the module), I believe that should live within the repo for the Terraform module in question. For Helix Core for example, depending on how much we want to automate we could even consider doing the following:

1. Default configuration would have pre-requisite to create the AMI and reference it in the module configuration before apply. This could be done using Packer or other means, and user could customize the image as they see fit.

2. Optional variable (e.g. enable_auto_image_creation) or something similar that can optionally use the local-exec provisioner to build the AMI and push to the target account on the user's behalf, then use that image for the deployment. We could determine if we want this image to persist in the account after a Terraform destroy (default behavior) or could potentially add another variable (e.g. disable_image_deletion_protection) which could use another local-exec to destroy the image on run of terraform destroy. I understand that using the local-exec provisioner is generally not recommended at scale, so this could be an optional experimental feature meant to simplify deployment. The only pre-requisite would be for the user to have Packer installed.

However, it is also a viable option to keep these Packer templates within the root toolkit repo and people can consume those as needed, and the module calls this out as a pre-requisite. I am unsure of how many potential options there could be for the AMI needed for Perforce, but I'm assuming there is likely a pretty generic one that the module can provide as a default, and users can customize this on their own if they want.

2. Does the Cloud Game Development Toolkit continue to exist as a collection of examples connecting these modules together in complex architectures? Do we publish it as a module as well (quickstart idea)?

Yeah, that's one option. I see two paths:

Option 1. The toolkit becomes as you mentioned a collection of examples that connect multiple modules together for purpose built solutions for more complex architectures. It can also contain helper resources such as the existing and future Packer templates and Ansible playbooks which ideally are used in combination with a linked Terraform module, but really could be used with an IaC tool besides Terraform, or even with ClickOps if a user desired this (though not recommended).

Option 2. The Cloud Game Development Toolkit itself becomes its own distinct Terraform module which uses the existing submodules (which become true submodules at that point). Conditional logic and optional parameters/variables would allow users to enable/disable certain functionality. The main downsides of this is that the project becomes more of a monolith, the code becomes substantially more complex, state becomes larger and potentially more complex to manage for end users, small changes could potentially hit service quota limits since so much infra could potentially be linked, and the blast radius is increased for resources that are deployed.

I believe option 1 is more viable in the longterm.

3. Do the examples in each module only pertain to that technology? In other words, where do we store a sample configuration that deploys Helix Core, Helix Swarm and the Helix Authentication Service? Maybe the solution is all three of these modules exist together and are released and versioned together?

Yes, I believe the examples in each module would just pertain to the specific technology. This makes the modules versatile, and the toolkit examples is what adds the Games lens for more complex implementations that use multiple of the modules together.

On Perforce - that's exactly what I was thinking. I believe Helix Core, Helix Swarm, and Helix Authentication Service should be contained within a single perforce module and conditional logic/optional parameters used to enable specific components, or not. For someone who just wants to use one part of the module, they would just modify the configuration to only enable those specific resources. However the assumption for the module would be that users would likely be deploying at least 2/3, in most cases all 3. The examples within this single Perforce module could demonstrate some of the customizations that are possible.

In terms of the MCP example, I just meant how the README links to the various versioned packages which from what I understand, can be separately installed and used as desired. This is what I am envisioning for the toolkit. Instead of python packages, pointing to the related repos on GitHub or modules in the Terraform Registry.

I recommend we use the Terraform Registry and the single git repo per module (this also simplifies the automated testing with GitHub actions). However, as an alternative to the separate Git repos/Terraform Registry:

• if the git tagging works for the submodule directories in a way that won't conflict with the git tags for the toolkit itself
• and/or if we're ok that versions for the modules would all change together even if changes were only made to a single module
• and we don't desire to host the modules on the Terraform Registry

Perhaps users could reference the directory for the modules and the specific git tag. Something like this:

Perforce

module "perforce" {
  source = git::https://github.com/aws-games/cloud-game-development-toolkit//modules/perforce?ref=v1.0.0)
}

TeamCity

module "teamcity" {
  source = git::https://github.com/aws-games/cloud-game-development-toolkit//modules/teamcity?ref=v1.0.0)
}

[kylesomers]

As more use cases and tools are identified and the number of modules grows, I agree that it will likely make sense to split these out into their own module repositories and manage them independently with separate communities of contributors that are supporting each of them. But that is not a near-term milestone in my opinion.

However, currently there is no way for a user of the toolkit to reference a specific version of any of the Terraform modules. For example, assuming there is a request to add SES email support for Perforce Helix Swarm. If and when this is added to the Perforce module, how can a user effectively use this new enhancement that was added to the module?

Currently this would involve subscribing to issue notifications and cherry picking commits as @henrykie described. This is a tradeoff we tolerate for now because we know that there will be breaking changes frequently to all of the modules until they are in a stable state (and we exit alpha). Definitely room for improvement here and would be addressed with module specific versioning.

[novekm]

Gotcha, makes sense. If the end goal is to eventually move towards individual git repos for the Terraform modules, I propose that we do the following:

1. Model Toolkit repo structure to align with future state

In the short term, I recommend that we model the toolkit repo structure to align with the future state of the modules eventually being in separate git repos. That way, if/when we transition in the future, the structure of the toolkit repo will largely stay identical. The only thing that would change is the modules directory would go away, and each module would git (pun intended) it's own repo.

This will make the potential transition easier in the future, and allow contributors to familiarize themselves with that structure so future contributions would largely stay the same. The only real difference they would experience is working in a separate git repo for the each of the modules, instead of within the /modules directory of the current repo. Ideally the all of the other components of the toolkit repo would stay generally the same even after the future transition.

Example directory structure

.
└── cloud-game-development-toolkit/
    ├── assets
    └── modules/
        ├── perforce/
        │   ├── examples/
        │   │   ├── basic/
        │   │   │   ├── main.tf
        │   │   │   ├── providers.tf
        │   │   │   └── # etc.
        │   │   └── existing-resources/
        │   │       ├── main.tf
        │   │       ├── providers.tf
        │   │       └── # etc.
        │   ├── tests/
        │   │   ├── 01_mandatory.tftest.hcl
        │   │   ├── 02_existing_resources.tftest.hcl
        │   │   └── # etc.
        │   ├── data.tf
        │   ├── iam.tf
        │   ├── local-exec.tf
        │   ├── local-file.tf
        │   ├── locals.tf
        │   ├── main.tf
        │   ├── outputs.tf
        │   ├── provider.tf
        │   ├── README.md
        │   ├── variables.tf
        │   ├── VERSION
        │   └── vpc.tf
        ├── jenkins/
        │   ├── examples/
        │   │   └── # etc.
        │   ├── tests/
        │   │   └── # etc.
        │   ├── main.tf
        │   ├── providers.tf
        │   └── # etc.
        ├── unreal/
        │   ├── horde/
        │   │   ├── examples/
        │   │   │   └── # etc.
        │   │   ├── tests/
        │   │   │   └── # etc.
        │   │   ├── main.tf
        │   │   ├── providers.tf
        │   │   └── # etc.
        │   ├── unreal-cloud-ddc/
        │   │   └── # etc.
        │   ├── tests/
        │   │   ├── 01_simple_build_pipeline.tftest.hcl
        │   │   └── # etc.
        │   └── examples/
        │       ├── simple-build-pipeline/
        │       │   ├── main.tf
        │       │   ├── providers.tf
        │       │   └── # etc.
        │       └── unreal-cloud-ddc-single-region/
        │           ├── main.tf
        │           ├── providers.tf
        │           └── # etc.
        ├── README.md
        ├── CONTRIBUTING.md
        └── # etc.

2. Consolidate the three Perforce modules into a single Perforce module

This is shown in the example directory structure above. I've already be working on this as a test. Essentially the single Perforce module could be used to deploy Helix Core, Helix Swarm, and Helix Authentication Service. Conditional logic would be used to optionally create infra. See this example main.tf file for the updates I've been working on:

main.tf

module "perforce" {
  source = "../../"
  # source = "aws-games/perforce/aws"
  # version = "v1.0.0"

  # SHARED
  fully_qualified_domain_name = "novekm.people.aws.dev"

  create_vpc           = true
  vpc_cidr_block       = "10.0.0.0/16"
  public_subnet_cidrs  = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnet_cidrs = ["10.0.3.0/24", "10.0.4.0/24"]


  # HELIX CORE
  helix_core_config = {
    resource_prefix = "helix-core"

    # Compute
    lookup_existing_ami      = false
    enable_auto_ami_creation = true
    ami_prefix               = "p4_al2023"

    instance_type         = "c6in.large"
    instance_architecture = "arm64"
    p4_server_type        = "p4d_commit" # required
    unicode               = false
    selinux               = false
    case_sensitive        = true
    plaintext             = false

    # Storage
    storage_type = "EBS"

    depot_volume_size    = 128
    metadata_volume_size = 32
    logs_volume_size     = 32

    # Networking
    lookup_existing_instance_subnet_id = false
    internal = false

    # Security & Auth
    create_helix_core_default_role = true

  }

  # HELIX SWARM
  helix_swarm_config = {
    # parameters

  }

  # HELIX AUTHENTICATION SERVICE
  helix_authentication_service_config = {
    # parameters

  }

In the module, infra is conditional created based on if the related components have the variable set, for example:

# only create this resource is var.helix_core_config exists and is not null
resource "aws_volume_attachment" "helix_core_depot_attachment" {
  count = var.helix_core_config != null ? 1 : 0
  # count       = var.helix_core_config.lookup_existing_instance_subnet_id == true ? 1 : 0
  device_name = "/dev/sdh"
  volume_id   = aws_ebs_volume.helix_core_depot[0].id
  instance_id = aws_instance.helix_core_instance[0].id
}

3. Support and document interim module versioning

For module versioning in the short term, users could continue to do as you mentioned - monitor GitHub issues and cherry pick commits. I'm not sure how this typically is achieved, but on the Terraform side, I think they should be able to use the core toolkit git tag that is used for the releases (e.g. v1.1.3-alpha) to achieve this. This would mean that all modules would would be linked to the same tag for now (which is tied to the toolkit releases), which I agree is probably fine while it's still in alpha.

For usage, it would look like:

Perforce

module "perforce" {
  source = git::https://github.com/aws-games/cloud-game-development-toolkit//modules/perforce?ref=v1.1.3-alpha
}

TeamCity

module "teamcity" {
  source = git::https://github.com/aws-games/cloud-game-development-toolkit//modules/teamcity?ref=v1.1.3-alpha
}

With the examples used, we should also start to reference the module sources in this way instead of using a local source (e.g. `source = "../../").

This also will ensure that for any enablement content we make before the transition, it is easy to update. All that would really change is the module source in the examples, and a some links in the docs

4. Standardize using EC2 Image Builder within all modules that need to manage container images

Outlined in #558. For now this would be another module within the toolkit /modules/ directory, but eventually could also be stored in its own git repo. Each module that needs to use it could either use the local source (e.g. source = "../moudles/ec2-image-builder") or use the git method of source = git::https://github.com/aws-games/cloud-game-development-toolkit//modules/ec2-image-builder?ref=v1.1.3-alpha.

[gabebatista]

I think both of you make a good point. Having the ability to version the individual modules makes a lot of sense to me, and having the modules available in TF registry would help discovery. My concerns are around how splitting the codebase across multiple repositories would affect the management of the project as whole. Being that the project team is very small at the moment, we need to minimize the effort required to keep the project organized. We should hold off on breaking out the modules until we exit alpha. At that point we should have a better handle on the project and will have automated many of the required tasks (which we can then replicate in the module specific repositories).

I think we could pilot this with the EC2 Image Builder module that @kylesomers is building if we wanted to, but where would that live? In aws-games? In aws-samples? Do we just avoid building our own module and contribute to the aws-ia/ec2-image-builder module? As Gabe mentioned we end up managing more overhead solely for the benefit of distributing these modules via TF registry and individual versioning. I'm not convinced the value is significant enough to end users to justify that.

Also, I'd like to see us remove the RFC issue type @gabebatista, and leverage GH discussions to avoid cluttering the backlog. I'll leave it up to you to determine if we close this and move the conversation elsewhere.

[novekm]

I agree that I could wait until we are out of alpha. On where to store the pilot EC2 Image Builder module (and the other Terraform modules in general) - I'd say it depends on how much autonomy we want to have. I don't have a strong preference on if the modules should live in the aws-games or aws-ia GitHub org if externalized. aws-samples has too many repos so I'd avoid that - I feel like it could just get buried there and lost in the sea of various unrelated repos. My recommendation would be to keep the modules in the aws-games git hub org (whether within the toolkit or in separate git repos), as this will allow us to still build with velocity and autonomy.

To help me understand the concerns better, what are the main complexities and additional overhead that you both envision? From what I gather so far it's:

• GitHub issue tracking
  • If externalizing the modules, these would each have their own GitHub issues, which would need to be linked to a centralized view for the toolkit, such as the GitHub Project that is being used for roadmap and sprint planning.
• Internal and External Contributor workflow
  • Instead of only having to think about a single repo (cloud-game-development-toolkit) they would have to interact with a variety of git repos
• Managing Releases
  • Instead of managing a single release and versioning for the toolkit, we would also be managing versions and releases for each of the separate modules

Am I off base or missing anything? In my mind, most of those above items can be resolved with user training/clear docs on contributing, and automation. Also, as discussed earlier - so far there aren't any net-new Terraform modules planned, so once moved to separate git repos and initially hosted, I don't envision that adding much either. Just incremental updates to those modules/repos based on GitHub issues. What I do think I'm hearing though is that the additional overhead is not really technical overhead, but project management overhead. If so, what are the key areas that become more difficult there?

On the point on end user value prop of TF registry and individual module versioning - I suppose my question is how do most Terraform developers consume modules, does this differ for game studios, and do we want to align the toolkit a typical consumption model or not?

From what I've seen, Terraform developers generally fall into one of three buckets in terms of module creation and/or consumption:

• [New, Intermediate, Advanced Users] - Find and consume Terraform Modules using the Terraform Registry
  • With this option, modules are consumed as is, while using the version to target specific versions of the modules. Any additional desired functionality is submitted in the form of a feature request via a GitHub issue raised against the underlying repo. This is the same process for Terraform Providers.
• [Intermediate, Advanced Users] - Build Custom Terraform Modules using starter code hosted somewhere
  • With this option, modules are still generally found using the public Terraform Registry. Users fork the repo and make their own customizations to the module as needed, instead of waiting on feature requests. Or, they simply copy and paste code from the source files, and use this as the foundation for their own custom Terraform Module that they store elsewhere.
• [Advanced Users] Build Terraform Modules from scratch
  • With this option, Terraform modules are developed using potentially no starter code

I'm not sure if this differs for game studios, but in my mind I'm assuming most users would do one of the first two options (option 1 even moreso for newer to intermediate Terraform developers). Either use the modules as-is by pointing to a public location such as the Terraform Registry, with the ability to easily manage versions, or fork one of the module git repos and work from there, potentially use the git module source that can reference a version based on git tags (e.g. `source = git::https://github.com/novekm/terraform-aws-perforce?ref=v1.0.0)

In the future, do we anticipate most users looking for ready-to-go modules that they can easily reference, or forking the modules and/or the cloning toolkit repo making extensive changes to them?

Either way, at the minimum I think it will help the end customers if they can reference a module version, even if in the short term, that is achieved by using the git tag for the toolkit itself (e.g. v1.1.3-alpha), and we start to use this in the examples. They could then use the release notes that would explain what changes were made within the modules, so they are aware that v1.1.4-alpha may have only impacted a single module, so they only need to bump the version in the source reference if they want the new changes just to that module. Perhaps that is already possible today.