500 errors while registering runners

[sidewinder12s]

I seem to be getting 500 errors somewhat randomly when I am running a playbook with serial: +1 (multiple runners are getting configured at the same time since it is running the tasks in parallel). I am not sure how prevalent it is, but I have seen it at least a few times with my GitLab instances though it is solved by just re-running the playbook.

Would be make sense to add some retry/wait logic around registering with GitLab or would it be better to just deal with it at the playbook level?

[drybjed]

Ansible only allows to set serial on the play level. I suppose that this could be set in the debops.gitlab_runner playbook by default if it works.

Did you try running with small amount of forks?

debops service/gitlab_runner --forks=1

Or, we could write a custom module to handle the registration and retry if necessary.

[sidewinder12s]

I didn't try the small amount of forks. I assume anything that limits the amount of runners that try to register at the same time would do as this seems like a GitLab Server issue though more of a corner case since most people won't be registering multiple runners at the same time. Maybe just a rescue block for the register to re-try/wait & retry?

[drybjed]

Sure, we could check if a rescue block would help. BTW, with what amount of hosts at once you see these issues? Are there any changes if you increase number of threads on the GitLab server?

[sidewinder12s]

This was with trying to register just 2 runners at the same time, just implementing the role separately from any other debops roles. I have not tried increasing the number of threads on the GitLab server since just re-running the playbook registered the second instance (So I've done no further troubleshooting 😄).

[sidewinder12s]

This could all also just be a bug in GitLab proper, but I haven't opened anything with them or looked at the production.log yet. I'll see if I can look at that next time I'm registering more runners.