Implement a front-door queue for build processing

[ecrupper]

Description

Using either Redis or another service, implement a front-door queue for processing webhooks from the SCM. This would involve separating the resource population / validation and the compiling in the PostWebhook API handler. The former would stay as in the handler while the latter would be moved to a separate Go routine (? - implementation details could change) that pops jobs off the queue.

This obviously is a large architectural refactor, but the skeletal structure is there: we already have a hooks table, so all compilation and validation errors are still able to be relayed to the user. Replying to GitHub with the build JSON is fairly pointless.

Value

This change in build processing would likely solve the long-standing concurrent webhooks issue, provided some extra atomic DB operations were added or even if we kept some basic retry logic. The strain or true concurrency (server has to accept and process requests) would be largely alleviated.

I think it's a prerequisite for merge queue to work without greatly exacerbating the above issue.

We will also be more aligned with what GitHub recommends for responsible webhook processing.

There are probably a great number of other benefits from pursuing this, but I'll call out a couple more:

• better build limit processing (rather than simply rejecting webhooks)
• data reliability improves (SCM delays for webhooks will not result in concurrency issues)
• less middleware bloat

Definition of Done

Vela works exactly the same from a user perspective, but underneath the hood the webhook processing has been split between intake and compilation/publishing.

Effort (Optional)

High.