Withdrawal scheduling

Withdrawals are created during POST /withdrawal, POST /rewards/:id/claim and POST /rewards/:id/give requests. The controllers for these endpoints respond with a scheduled Withdrawal.

This withdrawalDocument stored in the database that will have:

• a withdrawalId:number property if processing the job succeeded. This id is returned from the pool contract in the WithdrawPollCreated event and differs from the internal id used for reference to the document in the database..
• a failReason:string property if processing the job failed. Part of the job argument in the fail event callback. currentWithdrawalDocument is passed to the job attributes data in every processed withdrawal so there is a reference for it to be updated in case it fails.

Order Persistence

The withdrawal queue will be fetched with this mongodb query filter:

Withdrawal.find({ withdrawalId: { $exists: false } }, failReason: { $exists: false } || '' }).sort({ createdAt: -1 });

If a job fails because the current gas price obtained from the gas oracle is not below the configured env variable MAXIMUM_GAS_PRICE, the job will fail, but a failReason will not be set. Because of this the scheduled withdrawal will be part of the next run of the job processor and processed as first because of its createdAt property.

Job Processor Availability

• If one pool starts scheduling a lot of withdrawals other pools will notice a negative effect in terms of processing speed. One job processor per pool gas admin will solve the issue where one high usage pool blocks withdrawals for other pools

Concurrency

• If multiple instances of Agenda exist only one should be allowed to process jobs at a time. Agenda.maxConcurrency is set to 1 for this.
• If the job processor is locked newly scheduled withdrawals will not be part of the running job but instead scheduled (in order) for the next run.

Failed jobs

• If the job processor fails on a run it should store the failReason on the currently processed withdrawal
• If the failReason is not gas pricing related the withdrawal should not be part of the next run
• If the job processor complete event is cast (it is cast for both success and fail scenarios) it should run again in 5 seconds

High processing demand

• If withdrawal processing demand is high, and gas pricing is ok, the list of scheduled withdrawals will be sorted by the createdAt property and when the job runs it just starts processing them one by one until the job completes. Newly scheduled withdrawals will be processed in the run that start 5 seconds later.
• If withdrawal processing demand is high, but gas pricing is not ok, the job processor will fail and not touch the other scheduled withdrawals. It will run 5 seconds later and pick up from where it was until gas pricing is ok again.