Figure out how to build a custom Elastic Agent

[zmoog]

I need to make a change to Filebeat to properly set an input health status, and see it properly propagates to the Elastic Agent.

So I need to:

• Make a change to Filebeat
• Run this custom Filebeat version with Elastic Agent

[zmoog]

Here's what's worked better for me lately (raw notes from a conversation with a coworker):

Hmm, well the way I do it, I have everything in $HOME/go, and I have the repos at src/github.com/elastic/elastic-agent and src/github.com/elastic/beats and with that layout the command picks up the beats repo and builds from there. (Though sometimes it doesn't rebuild if there's already a tarball built in the beats repo, even if the source has changed. If changes aren't getting picked up try cleaning the beats build first to remove the old tarball.)

so if you:

cd ~/go/src/github.com/elastic/elastic-agent

elastic-agent$ EXTERNAL=false PLATFORMS=darwin/arm64 PACKAGES=TarGz mage dev:package

this builds beats from sources at ~/go/src/github.com/elastic/beats ?

it should, yes. I think that's the dev: part of the mage target, tho I'm not sure exactly what its assumptions are, but for me it works putting the repos under the default go path.

---

Let's give it a try.

[zmoog]

Clone or move the existing clone of Beats to ~/go/src/github.com/elastic/

I already have a Beats clone at ~/code/projects/elastic, so I'm moving this:

mv ~/code/projects/elastic/beats ~/go/src/github.com/elastic/

# walk into the Beats repo and pull the latest changes from main
cd ~/go/src/github.com/elastic/

git switch main
git pull

Current latest commit hast is 1f21f0 (just for the records).

[zmoog]

Making a change to the azure-eventhub input v2 by adding a simple log statement:

diff --git a/x-pack/filebeat/input/azureeventhub/v2_input.go b/x-pack/filebeat/input/azureeventhub/v2_input.go
index 84755ff388..55652dee52 100644
--- a/x-pack/filebeat/input/azureeventhub/v2_input.go
+++ b/x-pack/filebeat/input/azureeventhub/v2_input.go
@@ -79,6 +79,8 @@ func (in *eventHubInputV2) Run(
        inputContext v2.Context,
        pipeline beat.Pipeline,
 ) error {
+       in.log.Info("Starting a custom Azure Event Hub input v2")
+
        var err error

        ctx := v2.GoContextFromCanceler(inputContext.Cancelation)

And let's double-check it shows up in the Filebeat logs using the following config:

# x-pack/filebeat/modules.d/azure.yml
- module: azure
  # All logs
  activitylogs:
    enabled: true
    var:
      # eventhub name containing the activity logs, overwrite he default value if the logs are exported in a different eventhub
      eventhub: "${EVENTHUB_NAME}"
      # consumer group name that has access to the event hub, we advise creating a dedicated consumer group for the azure module
      consumer_group: "$Default"
      # the connection string required to communicate with Event Hubs, steps to generate one here https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string
      connection_string: "${EVENTHUB_CONNECTION_STRING}"
      # the name of the storage account the state/offsets will be stored and updated
      storage_account: "${EVENTHUB_STORAGE_ACCOUNT}"
      # the name of the storage account container you would like to store the offset information in.
      # storage_account_container: "${EVENTHUB_STORAGE_ACCOUNT}"
      # the storage account key, this key will be used to authorize access to data in your storage account
      # storage_account_key: "${STORAGE_ACCOUNT_KEY}"
      storage_account_connection_string: "${EVENTHUB_STORAGE_ACCOUNT_CONNECTION_STRING}"
      processor_version: "v2"

[zmoog]

Here's the log from plain Filebeat:

{
  "log.level": "info",
  "@timestamp": "2025-07-21T16:36:02.839+0200",
  "log.logger": "input.azure-eventhub",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.(*eventHubInputV2).Run",
    "file.name": "azureeventhub/v2_input.go",
    "file.line": 82
  },
  "message": "Starting a custom Azure Event Hub input v2",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

[zmoog]

Build the Elastic Agent

Not it's the EA turn. Let's walk into the EA clone and try building it.

cd ~/go/src/github.com/elastic/elastic-agent

Build the Elastic Agent:

$ EXTERNAL=false PLATFORMS=darwin/arm64 PACKAGES=docker mage dev:package
--- Package Elastic-Agent
Package "agentbeat-9.2.0-darwin-aarch64.tar.gz" does not exist on path: /Users/zmoog/go/src/github.com/elastic/beats/x-pack/agentbeat/build/distributions/agentbeat-9.2.0-darwin-aarch64.tar.gz
--- Package /Users/zmoog/go/src/github.com/elastic/beats/x-pack/agentbeat
>> Packaging agentbeat that includes [auditbeat filebeat heartbeat metricbeat osquerybeat packetbeat]
>> Changing into auditbeat directory
>> Executing mage update for auditbeat
Generated fields.yml for auditbeat to /Users/zmoog/go/src/github.com/elastic/beats/x-pack/auditbeat/fields.yml
>> Building auditbeat.yml for linux/amd64
>> Building auditbeat.reference.yml for linux/amd64
>> Building auditbeat.docker.yml for linux/amd64
>> Changing into filebeat directory
>> Executing mage update for filebeat
No fields files for module add_nomad_metadata
No fields files for module aws_vpcflow
.
.
.
[cut]
.
.
.
Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.24.5-darwin-arm64-debian11
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[CC=oa64-clang, CXX=oa64-clang++, GOARCH=arm64, GOARM=, GOOS=darwin, GOTOOLCHAIN=local, PLATFORM_ID=darwin-arm64]
--- Package artifact
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
package ran for 10m8.623109125s

[zmoog]

This command doesn't produce a docker image 🤔 — I'm probably missing something here.

[zmoog]

I tried a different config from #35 (comment)

DEV=true \
SNAPSHOT=true \
PLATFORMS="linux/arm64" \
PACKAGES="docker" \
mage -v package

And finally I got a container image for my custom Elastic Agent 9.2.0-SNAPSHOT:

➜ docker images | grep "9\.2"
docker.elastic.co/elastic-agent/elastic-agent-complete             9.2.0-SNAPSHOT                                                            5f5ed715eae4   4 minutes ago   4.63GB
docker.elastic.co/beats-ci/elastic-agent-cloud                     9.2.0-SNAPSHOT                                                            9efebe0c6dfb   5 minutes ago   2.65GB
docker.elastic.co/elastic-agent/elastic-otel-collector             9.2.0-SNAPSHOT                                                            f3c944f5f48a   5 minutes ago   2GB
docker.elastic.co/elastic-agent/elastic-otel-collector-wolfi       9.2.0-SNAPSHOT                                                            591d544163f6   5 minutes ago   1.87GB
docker.elastic.co/elastic-agent/elastic-agent-complete-wolfi       9.2.0-SNAPSHOT                                                            e341d9662b58   6 minutes ago   4.1GB
docker.elastic.co/elastic-agent/elastic-agent                      9.2.0-SNAPSHOT                                                            51bf5b780eb4   8 minutes ago   2.01GB
docker.elastic.co/elastic-agent/elastic-agent-ubi                  9.2.0-SNAPSHOT                                                            031e8a63acc5   8 minutes ago   2.01GB
docker.elastic.co/elastic-agent/elastic-agent-wolfi                9.2.0-SNAPSHOT                                                            338a13d72f74   8 minutes ago   1.88GB

[zmoog]

Set up a test cluster

I'm using an existing serverless project, but you can use any Elastic stack.

You need to:

1. Create an agent policy and install the Azure Logs v2 integration
2. Enrol the custom Elastic Agent

Install the Azure Logs v2 integration

Visit Fleet, and create an empty agent policy and install the Azure Logs v2 integration.

The change we want to test is on the processor v2, so you need to enable the processor v2 in the Advanced options section:

Enroll the custom Elastic Agent

In Fleet, start enrolling a new agent. Take note of the following information from the panel:

• Fleet URL
• Fleet enrollment token

[zmoog]

Run the custom Elastic Agent

Fleet is now waiting for the new agent to enroll. Let's enroll a new agent using a container with one of the images we built earlier:

docker run --rm \
  -e FLEET_URL="https://[redacted].fleet.eastus.azure.elastic.cloud:443" \
  -e FLEET_ENROLL=true \
  -e FLEET_ENROLLMENT_TOKEN="[redacted]" \
  -i docker.elastic.co/elastic-agent/elastic-agent:9.2.0-SNAPSHOT

[zmoog]

Check the log

Search for the agent logs in Fleet or discover: