[WIP] Update minimum required Terraform version and resolve various deprecation notices

[strangeways]

In order to update the AWS provider version (see the note at the bottom), we need to use a newer version of Terraform (1.4.5). However, this version doesn't support the deprecated hashicorp/template provider on non-Intel Macs. (See discussions here and here.) A temporary workaround is to install the Intel version of Terraform, but this is annoying to set up, and the deprecated code needs to be updated anyway.

This PR makes the following changes:

• Replace the use of the template_file provider with the built-in templatefile function.
• Update S3 bucket configurations to use the new format, which splits the configuration into different resource blocks.

Some of these changes already existed upstream. The others were added in controlshift#41

Due to the change in how the S3 bucket resources are defined, we need to run the following import commands BEFORE terraform apply so that Terraform knows these resources already exist (otherwise it will try to delete and recreate them). Note: GLUE_SCRIPTS_BUCKET_NAME and MANIFEST_BUCKET_NAME must be replaced with the actual S3 bucket names.

terraform import aws_s3_bucket_ownership_controls.glue_resources GLUE_SCRIPTS_BUCKET_NAME
terraform import aws_s3_bucket_acl.glue_resources GLUE_SCRIPTS_BUCKET_NAME
terraform import aws_s3_bucket_server_side_encryption_configuration.glue_resources GLUE_SCRIPTS_BUCKET_NAME
terraform import aws_s3_bucket_lifecycle_configuration.glue_resources GLUE_SCRIPTS_BUCKET_NAME
terraform import aws_s3_object.signatures_script GLUE_SCRIPTS_BUCKET_NAME/production/signatures_job.py

terraform import -provider=aws.controlshift aws_s3_bucket_ownership_controls.manifest MANIFEST_BUCKET_NAME
terraform import -provider=aws.controlshift aws_s3_bucket_acl.manifest MANIFEST_BUCKET_NAME
terraform import -provider=aws.controlshift aws_s3_bucket_server_side_encryption_configuration.manifest MANIFEST_BUCKET_NAME
terraform import -provider=aws.controlshift aws_s3_bucket_lifecycle_configuration.manifest MANIFEST_BUCKET_NAME

The import commands update the Terraform state file that is saved in S3 and which represents the current state of the deployed infrastructure. After running the imports, run terraform plan and check the following:

• There should be 0 to destroy
• The aws_s3_bucket resources should show ~ update in-place

If you want to examine the state file itself, you can use terraform state list to list the resources addresses and terraform state show [RESOURCE_ADDRESS] to show the state details for the given resource.

--

Items to be implemented separately due to their complexity:

• Sync any changes from upstream that we want to include. In particular, the change to the signatures job template file (and the related changes in glue_job.tf) would be nice to have.
• Update the AWS provider to "~> 5.26.0" and the Node.js version on the four lambdas. This may be tricky; see details below:
  • The AWS provider version needed to be updated in order to support a newer version of Node.js (see PR Update AWS version to one with support for nodejs20.x #19). The lambdas in this repo are using 16.x, which AWS has deprecated. Updates will no longer be allowed after July 1, 2026.
  • Previous PRs attempted to update to Update Node to version 20 but then downgraded back to version 16. It seems that the lambdas need to be updated to a new version of the AWS SDK.
  • It's also not clear how Update Redshift loader version #21 is related. It updates the AWSLambdaRedshiftLoader from version 2.7.8 to version 2.8.3, but this repo lists 2.7.8 as the latest release so it might not be the correct source.