Agent segementation fault when starting with backlog of jobs on Linux

[rubyroobs]

I unfortunately don't have any reproducible steps and we resolved this by removing the registered agent and retrying, but in case this is a regression or other people experience this I wanted to make an issue to track it.

Description

The OpenBAS agent service segmentation faults within a few seconds of starting after beginning to execute a backlog of jobs (with the number of jobs succeeding before the crash varying each time).

Environment

1. OS (where OpenBAS server runs): Debian 12 bookworm
2. OpenBAS version: "Starting OpenBAS agent 1.12.2"
3. OpenBAS client: Frontend (?)
4. Other environment details: occurs after reinstalling the client by re-running the curl/shell installer

Actual Output

From journalctl:

Feb 19 00:48:39 hostname systemd[1]: Starting openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:39 hostname systemd[1]: Started openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:41 hostname systemd[1]: Stopping openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:41 hostname systemd[1]: openbas-agent.service: Deactivated successfully.
Feb 19 00:48:41 hostname systemd[1]: Stopped openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:41 hostname systemd[1]: openbas-agent.service: Consumed 1.685s CPU time.
Feb 19 00:48:41 hostname systemd[1]: Starting openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:41 hostname systemd[1]: Started openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:43 hostname systemd[1]: Stopping openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:43 hostname systemd[1]: openbas-agent.service: Deactivated successfully.
Feb 19 00:48:43 hostname systemd[1]: Stopped openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:43 hostname systemd[1]: openbas-agent.service: Consumed 1.580s CPU time.
Feb 19 00:48:43 hostname systemd[1]: Starting openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:43 hostname systemd[1]: Started openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:45 hostname systemd[1]: Stopping openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:45 hostname systemd[1]: openbas-agent.service: Deactivated successfully.
Feb 19 00:48:45 hostname systemd[1]: Stopped openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:45 hostname systemd[1]: openbas-agent.service: Consumed 2.031s CPU time.
Feb 19 00:48:45 hostname systemd[1]: Starting openbas-agent.service - OpenBAS Agent...
Feb 19 00:48:45 hostname systemd[1]: Started openbas-agent.service - OpenBAS Agent.
Feb 19 00:48:45 hostname systemd[1]: openbas-agent.service: Main process exited, code=killed, status=11/SEGV
Feb 19 00:48:45 hostname systemd[1]: openbas-agent.service: Failed with result 'signal'.

Additional information

Screenshots (optional)

[antoinemzs]

Hi @rubyroobs thanks for the filing. I'm curious about the fact that the agent is restarting several times a second, this looks like there may be a problem with it. The intended design is that the agent is a long lived daemon which pings the OpenBAS server periodically. One reason for it to restart spontaneously is that the server is proposing an updated agent binary and the agent daemon self-updates. One possible error case which would potentially create the traces above is that the self-update fails and when restarting, the agent daemon tries another self-update (and fails in a loop).

You mention that you reinstalled the agent and that the issue did not occur again since, I suspect that whatever which may have been wrong wrt the agent's install might have been fixed then.

Was the agent already installed, and the OpenBAS server updated after that fact? Or was the agent installed using the command proposed by the current OpenBAS server's version?

[rubyroobs]

Hi @antoinemzs

AFAIK we were not manually continuously restarting the agent, and I didn't see anything in the logs about it trying to update. We actually did try to update it/re-install it by re-running the previously ran command when it crashed, and may have updated OpenBAS versions between these. Based on the timing with the agent logs I do think it was occurring while processing jobs (unless the update process itself was a job?). What seemed to fix it was deleting the agent in the OpenBAS UI, recreating a new one and then installing again via the new proposed command.

I'm fairly confident in this but unfortunately I don't have logs to say with certainty for anything, but if we run into this again I will try to more carefully collect the logs and check whether updating was what resolved the issue etc.

[antoinemzs]

The agent's autoupdate is indeed itself a special-case agent job automatically created when the agent service (running on the asset) calls the register server endpoint, and then the server may queue up the update job if it finds that there is an agent update available. The agent executes the job whenever it picks it up from the queue, which might be consistent with the observed behaviour on your setup.

IIRC there might not be log traces of the autoupdate process, but you may find a queued up job in the asset_agent_jobs table. However if the issue is not reproducible on your end, you might not find much standing out in the database.