Audit test pipelines

[cachedout]

This is a master tracking issue issue for auditing which E2E test pipelines need to remain enabled.

Beats CI pipelines

Pipeline	Main Health	Triggers	Stakeholders	Issue(s)	Removal planned
Docker images	⭕ Stale	None	Robots [@cachedout and @kuisathaverat ]		❌
Fleet E2E	🔴 Broken	Daily build	Fleet [@joshdover]	elastic/elastic-agent#1174
Observability Helm Charts	🟢 Healthy	Daily build	Robots [@cachedout and @kuisathaverat ]	Issue located in private repo	✅
K8S Autodiscover	🟡 Flakey	Daily build	Cloud Native Monitoring [@gizas]
Observability MacOS	🔴 Broken	Daily build	Elastic Agent [@cmacknz and @jlind23 ]	https://github.com/elastic/ci/issues/705
Fleet Server	⭕ Stale	None	Fleet [@joshdover ]	elastic/fleet-server#1927
Fleet UI	⭕ Stale	None	Fleet and Integrations [@kpollich ]

Fleet CI pipelines

Pipeline	Main Health	Triggers	Stakeholders	Issue
Pipeline helper	🔴 Broken	Push to main; PR labeled	Elastic Agent[@cmacknz and @jlind23 ]	elastic/elastic-agent#1174

⚠️ If you are listed as a stakeholder, we would like to know the following:

1. Should the pipeline be removed from the CI or should it remain?
   1.1 If the pipeline remains and is broken, what is the link to an issue tracking a fix?
   1.2 If the pipeline should remain, how is it monitored by the team to ensure that build artifacts are not produced when the tests fail?

Next steps

Proposed pipeline criteria

I am proposing that we remove all pipelines which do not meet any of the following criteria:

1. Necessary for the ongoing health of the E2E test suite itself
2. Used by a product team as a quality gateway. Concretely, this means that a failing test blocks a PR from being merged or a build artifact from being produced.
3. Exist to ensure the quality of a supported product.

Timeline

1. All existing E2E pipelines have stakeholders assigned no later than: October 1, 2022
2. All stakeholder agree upon proposed pipeline criteria no later than: October 20, 2022
3. Non-confirming pipelines will be removed from Jenkins and code will be removed from the E2E test suite beginning on: Nov 1st, 2022

Related efforts

There is a separate effort to try and reduce the scope of E2E testing back to a point where stability can be maintained, but it is limited to tests for the Agent. That effort can be found here: elastic/elastic-agent#1174

[v1v]

For the Macos Daily -> it was originally implemented in #2626, and using the Orka ephemeral workers, and superseded #2336

the error is something the @elastic/ci-systems might need to help with:

[2022-09-28T04:58:54.763Z] + .ci/scripts/deployment.sh create
[2022-09-28T04:58:54.887Z] Cloning into '.obs'...
[2022-09-28T04:58:55.095Z] Host key verification failed.
[2022-09-28T04:58:55.095Z] fatal: Could not read from remote repository.
[2022-09-28T04:58:55.095Z] 
[2022-09-28T04:58:55.095Z] Please make sure you have the correct access rights
[2022-09-28T04:58:55.095Z] and the repository exists.

IIUC, the recent upgrade in the CI controllers added a host key verification by default, we reported this in the past and it was partially fixed since we dont' see the below error but a new one:

but the error now happens in a subsequent stage to clone a private repository -- see the above console log

It worked in the past

[kuisathaverat]

Docker images generated the Systemd Docker images used in the e2e tests, probably we are the stakeholders.

[cachedout]

@v1v Thanks, that helps. I'm also trying to figure out what it actually does so that I can figure out how the stakeholders should be. I'm code-diving right now a bit to try and get a sense of that.

[kuisathaverat]

Observability Helm Charts can be removed

[cachedout]

@kuisathaverat Thanks! Regarding the Docker images -- that pipeline hasn't been executed for over a year. Does it still need to exist?

[kuisathaverat]

Does it still need to exist?

It is the only way to generate those images, when they change should be executed. These images are for making a test on installation on a systems environment. The main changes that can have are bumping the systems version or the Linux version.

[cachedout]

@cmacknz and @jlind23 Are you tracking any issues for the flakiness in the K8s Autodiscover pipeline?

[v1v]

@v1v Thanks, that helps. I'm also trying to figure out what it actually does so that I can figure out how the stakeholders should be. I'm code-diving right now a bit to try and get a sense of that.

There was an original request to test on MacOS, for such, it was initially attempted with the AWS MacOS, but it was declined for vary reasons:

1. Cost, IIRC, machines will be created and pay for 24 hours minimal, see Add Mac OS12 as part of the platform tested #2336 (comment)
2. Implementation, the Ansible ec2 integration didn't work well , see Add Mac OS12 as part of the platform tested #2336 (comment)
3. Ephemeral Orkas were available. see Add Mac OS12 as part of the platform tested #2336 (comment)

I guess the stakeholder might be @jlind23 as he was the original requester for the MacOS in AWS

[jlind23]

@cachedout this is the issue we will use for the first half of 8.6. @AndersonQ is already assigned to this and will closely work with you in order to get back to a better place.

[cachedout]

@jlind23 That link seems wrong? :)

[jlind23]

Sorry, this one - elastic/elastic-agent#1174

[cmacknz]

@cmacknz and @jlind23 Are you tracking any issues for the flakiness in the K8s Autodiscover pipeline?

No, it may make sense to follow up with the Observability Cloudnative monitoring team to see if they have interest in fixing these tests faster than the agent team can get to them. They have done the majority of the recent work for autodiscovery features in agent.