How to deploy an unreleased version of ESF (Terraform install method)

[zmoog]

I worked on upgrading the Python runtime to 3.12 (elastic/elastic-serverless-forwarder#874 ) and now I want to test it using Terraform.

How to install ESF using Terraform:

• https://github.com/elastic/elastic-serverless-forwarder/tree/main/dev-corner/how-to-test-locally

[zmoog]

Step 0: Build your lambda package

In your local clone of https://github.com/elastic/elastic-serverless-forwarder

make package

Step 1: Build your dependencies zip file

# check your task version, older version may fail — it wasn't working for me with task 3.26
➜ task --version
3.43.3

See https://taskfile.dev/docs/installation for the latest install instructions.

➜ du -sh local_esf.zip 
 18M    local_esf.zip


➜ unzip -l local_esf.zip
Archive:  local_esf.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  05-05-2025 11:00   orjson-3.10.16.dist-info/
      965  05-05-2025 11:00   orjson-3.10.16.dist-info/RECORD
        0  05-05-2025 11:00   orjson-3.10.16.dist-info/licenses/
    10847  05-05-2025 11:00   orjson-3.10.16.dist-info/licenses/LICENSE-APACHE
     1023  05-05-2025 11:00   orjson-3.10.16.dist-info/licenses/LICENSE-MIT
      104  05-05-2025 11:00   orjson-3.10.16.dist-info/WHEEL
        0  05-05-2025 11:00   orjson-3.10.16.dist-info/REQUESTED

[...]

     3589  04-16-2025 18:18   ../../shippers/__pycache__/factory.cpython-312.pyc
     1522  04-16-2025 18:18   ../../shippers/__pycache__/shipper.cpython-312.pyc
    16405  04-17-2025 13:15   ../../shippers/__pycache__/es.cpython-312.pyc
      609  04-16-2025 18:18   ../../shippers/__pycache__/__init__.cpython-312.pyc
     3263  10-07-2024 06:29   ../../shippers/factory.py
     5562  07-25-2024 12:27   ../../shippers/logstash.py
    15803  04-17-2025 13:06   ../../shippers/es.py
     2722  04-27-2023 10:32   ../../shippers/composite.py
---------                     -------
 32967695                     4335 files

[zmoog]

Step 2: Run ESF terraform

Copy the local_esf.zip in your local clone of elastic/terraform-elastic-esf repo:

➜ cp local_esf.zip ../../../terraform-elastic-esf/

Edit the esf.tf file to use the local file instead of the release version:

diff --git a/esf.tf b/esf.tf
index d431742..b806160 100644
--- a/esf.tf
+++ b/esf.tf
@@ -8,7 +8,8 @@
 ###### Elastic Serverless Forwarder
 locals {
   dependencies-bucket-url = "http://esf-dependencies.s3.amazonaws.com"
-  dependencies-file       = "${var.release-version}.zip"
+  # dependencies-file       = "${var.release-version}.zip"
+  dependencies-file = "local_esf.zip"
 
   attach_network_policy = (var.vpc != null ? true : false)
 
@@ -136,18 +137,19 @@ resource "aws_s3_object" "config-file" {
   depends_on = [aws_s3_bucket.esf-config-bucket]
 }
 
-resource "terraform_data" "curl-dependencies-zip" {
-  provisioner "local-exec" {
-    command = "curl -L -O ${local.dependencies-bucket-url}/${local.dependencies-file}"
-  }
-}
+# resource "terraform_data" "curl-dependencies-zip" {
+#   provisioner "local-exec" {
+#     command = "curl -L -O ${local.dependencies-bucket-url}/${local.dependencies-file}"
+#   }
+# }
 
 resource "aws_s3_object" "dependencies-file" {
   bucket = local.config-bucket-name
   key    = local.dependencies-file
   source = local.dependencies-file
 
-  depends_on = [aws_s3_bucket.esf-config-bucket, terraform_data.curl-dependencies-zip]
+  # depends_on = [aws_s3_bucket.esf-config-bucket, terraform_data.curl-dependencies-zip]
+  depends_on = [aws_s3_bucket.esf-config-bucket]
 }

[zmoog]

Deploy

In the elastic/terraform-elastic-esf repo.

Config file:

lambda-name            = "mbranca-esf-py312"
release-version        = "not-used" # See https://github.com/elastic/elastic-serverless-forwarder/tags
# config-file-bucket     = "my-esf-bucket" # Uncomment if s3 bucket pre-exists
# config-file-bucket     = "mbranca-esf-configs" # Uncomment if s3 bucket pre-exists
aws_region               = "eu-west-1"
# config-file-local-path = "./config.yaml" # Uncomment if local config path is used
inputs = [
  {
    type = "s3-sqs"
    id   = "arn:aws:sqs:eu-west-1:[redacted]:mbranca-esf-notifications"
    outputs = [
      {
        type = "elasticsearch"
        args = {
          elasticsearch_url  = "[redacted]"
          api_key            = "[redacted]"
          es_datastream_name = "logs-esf.mbranca-default"
        }
      }
    ]
  }
]

Deploy:

terraform init
terraform apply

[zmoog]

Test

Send a notification to trigger the ESF execution:

aws sqs send-message \
    --queue-url https://sqs.eu-west-1.amazonaws.com/123/mbranca-esf-notifications \
    --message-body '{"Records":[{"awsRegion":"eu-west-1","eventTime": "2024-01-31T19:06:30.680Z","eventName":"ObjectCreated:Put","eventSource":"aws:s3","eventVersion":"2.1","s3":{"bucket":{"arn":"arn:aws:s3:::mbranca-esf-data","name":"mbranca-esf-data"},"object":{"key":"123_CloudTrail_eu-west-1_20240718T0005Z_Uq6vutkBDErYhBue.json.gz"}}}]}'

UPDATE

The test failed. Unfortunately there are a few problem to address:

• elastic/terraform-elastic-esf: the terraform file hard codes the Python 3.9 runtime
• elastic/elastic-serverless-forwarder: Taskfile only works if you run it on the linux/amd64 platform, while I'm on macOS/darwin (aarm64).

elastic/terraform-elastic-esf

For the hard-coded runtime version, I made this additional change:

@@ -157,7 +159,7 @@ module "esf-lambda-function" {
 
   function_name = var.lambda-name
   handler       = "main_aws.lambda_handler"
-  runtime       = "python3.9"
+  runtime       = "python3.12"
   architectures = ["x86_64"]
   timeout       = var.lambda-timeout
 

elastic/elastic-serverless-forwarder

I re-built the local-esf.zip for the linux/amd64 platform (I copied the steps from .github/workflows/releases-production.yml workflow) by running a quick Makefile:

local-zip:
	apt-get update && apt-get install -y zip && \
	cd /esf && \
	rm -rf deps && \
	mkdir deps && \
	pip3 install -r requirements.txt -t deps && \
	cd deps && zip -r ../local_esf.zip . && \
	cd .. && \
	zip -r local_esf.zip main_aws.py && \
	zip -r local_esf.zip handlers && \
	zip -r local_esf.zip share && \
	zip -r local_esf.zip storage && \
	zip -r local_esf.zip shippers

and running:

docker run \
  --platform linux/amd64 \
  -v .:/esf \
  -v ./Makefile:/Makefile python:3.12 make local-zip

With these changes, the lambda deployed with Terraform is finally able to start: