Elastic agent fails in AIT when TLS is enabled

[jamiesmith]

Certain parts of alerting and fleet require TLS to be enabled to work, and when it is the startup of the Elastic Agent container fails with this error:

Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://kibana:5601/api/fleet/setup": EOF. Response: 

This is (essentially) the command that I use to start up my environment with TLS enabled for the stack:

./scripts/compose.py start \
    --with-opbeans-dotnet01 \
    --with-opbeans-go01 \
    --with-opbeans-java01 \
    --with-opbeans-node01 \
    --with-opbeans-python01 \
    --with-opbeans-ruby01 \
    --with-elastic-agent \
    --elasticsearch-enable-tls \
    --kibana-enable-tls \
    --opbeans-elasticsearch-url https://elasticsearch:9200 \
    --elasticsearch-heap 16g \
    --all \
    7.13.0

Notice that I override the --opbeans-elasticsearch-url in there. We likely need a similar kind of --elastic-agent-kibana-url with that, and under the covers maybe need to override the certs and CA, similar to the APM server, here's a partial snippet from the generated compose file:

"apm-server.kibana.ssl.certificate_authorities=[\"/usr/share/apm-server/config/certs/stack-ca.crt\"]"

Thanks!

[kuisathaverat]

--elasticsearch-enable-tls and --kibana-enable-tls are correctly managed now.

[jamiesmith]

Should these work with the --with-elastic-agent option? Can you provide an example command?

I tried adding them to my start and the agent container dies.

This command:

./scripts/compose.py start  \
    --with-elastic-agent \
    --no-verify-server-cert \
    --elasticsearch-enable-tls \
    --kibana-enable-tls \
    --elasticsearch-heap 16g \
    --all \
    7.15.0-SNAPSHOT

Generates:

Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://kibana:5601/api/fleet/setup": EOF. Response:

So I added --elastic-agent-kibana-url https://kibana:5601 to the startup command, which is closer, but changes the error to:

Kibana Fleet setup failed: http POST request to https://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "https://kibana:5601/api/fleet/setup": x509: certificate signed by unknown authority. Response: 

Thanks

[kuisathaverat]

I just tested this command and check that the compose file uses https URLS, then I have checked that there is an Elastic Agent attached and online, Where do you see this error? could it be the fleet endpoint config what it is wrong?

./scripts/compose.py start  \
    --with-elastic-agent \
    --elasticsearch-enable-tls \
    --kibana-enable-tls \
    8.0.0

[jamiesmith]

How long did you let yours run?

I am guessing that Elastic Agent likely failed after about a minute in your execution (or maybe 30 attempts), trying to talk to Kibana:

Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://kibana:5601/api/fleet/setup": EOF. Response: 
Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://kibana:5601/api/fleet/setup": EOF. Response: 
Error: http POST request to http://kibana:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://kibana:5601/api/fleet/setup": EOF. Response: 
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.15/fleet-troubleshooting.html

That same error from both 7.15.0 and 8.0.0, so it's not really running

[jamiesmith]

(that was why I added the --elastic-agent-kibana-url https://kibana:5601, which gets it closer, but then agent complains about the self-signed cert)