HTTP Transactions: Why not queue the jobs after the user record is successfully created?

[batiste93]

Apologies if this question doesn't belong here.

I'm not sure I understand the need to create a staging table for the jobs. Why not just wait for the user record transaction to succeed then enqueue the job?

REF: https://github.com/brandur/sorg/blob/master/content/articles/http-transactions.md#background-jobs-background-jobs

Any clarification is appreciated.

Thanks

[brandur]

Hi @batiste93,

The reason is that having some kind of hook on transaction commit still leaves you open to occasionally losing data.

Say for example that you've got a web worker running. It creates the user record, commits the transaction, prepares create jobs, but then suddenly dies before it was able to do so. There are no other failsafes to ensure recovery in that situation, so that job is never run.

By using the staged jobs idea, we ensure that if a user record is created then a job is also created. Either the process that created those records or the job runners can die and we still guarantee that the job gets run, even in the most unlikely edge failures.

Hopefully that clarifies things!

[eporomaa]

Thanks for the series on Idenmpotency!

Somewhat related question,

Assuming that the "enqueuer" runs on some interval with exponential back-off when no jobs are found. Given that we don't want the "enqueuer" to "be late" once a job is committed. Would it be wise to notify the "enqueuer", resetting it's exponential back-off, in the application layer after the transaction has committed?

In our case we want to:

1. minimize the load on db when no jobs are queued,
2. react as fast as possible when they are queued
3. minimize functional logic in db

Thoughts?

[brandur]

Hey @eporomaa,

Yeah, that totally makes sense to me just as long as you have a good way to implement it. You'd probably have to have something like a post-commit on every transaction, and there is some possibility that could get expensive. If you have a few transactions committing a minute and notifying on them, then that's no problem, but if you have tens of thousands of them all trying to send notifies, then that gets more problematic.

I probably wouldn't bother as long as long as I could assume semi-regular load in my application. An alternative strategy to ensure semi-regular throughput on applications without much load might be to just reduce the maximum exponential backoff to just a few seconds — if you have the enqueuer running every few seconds the DB load should be ~negligible.

[eporomaa]

@brandur Makes sense, thx! :)