Releases: Betterment/delayed
v2.0.2 - improved index creation lock-handling
What's Changed
- fix: Allow statement timeouts on index creation to retry (within migration wait timeout) by @smudge in #86
Full Changelog: v2.0.1...v2.0.2
v2.0.1
v2.0.0 - The "Database-Driven Development" Release!
What's Changed
Numerous changes focused on index coverage and query performance went into this release.
At a high level, this release consists of:
- New optional-but-encouraged DB indexes (available via
rake delayed:install:migrations) that improve index coverage of all queries. - Adjustments to queries to improve selectivity and reduce cost (# of scanned/filtered rows).
- First class support for HOT updates in PostgreSQL (during job pickup query).
- Daylight savings time fixes in non-UTC time (
:local) contexts. (#81) - Job timeouts can no longer be rescued from within the
performmethod as aStandardError(#66)
YMMV (and will depend heavily on the contents of your queue and the number of workers you run). In real-world at-scale testing against a PostgreSQL (RDS Aurora) backed queue, in an environment where the table had many millions of future-scheduled and/or failed rows (i.e. "non-claimable" rows), improvements were observed across the board:
- The worker pickup query saw a 100-1000x query time improvement.
- The "monitor" queries saw on net a 10-100x improvement.
- Overall CPU usage and disk I/O of an active (but not back-logged) PostgreSQL queue dropped significantly.
The full list of changes is as follows:
- test: Add "golden" tests for worker/monitor SQL by @smudge in #61
- refactor: Add 'lock_timeout' & clarify that 'max_run_time' is a process-wide config by @smudge in #62
- refactor: Remove 'ready_scope' and improve remaining scopes. by @smudge in #63
- fix(tests): timing issue on test flake by @smudge in #65
- fix: max_run_time timeout error should not be rescuable as a StandardError by @smudge in #66
- test: Use snapshots for easier SQL testing/iteration. by @smudge in #67
- test: Normalize DB versions used in CI by @smudge in #68
- test: Snapshot test all EXPLAIN query plans by @smudge in #69
- feat(perf): New indexes for job pickup & monitoring by @smudge in #70
- fix: Ensure that upsert_index is reversible by @smudge in #72
- fix(monitor): exclude failed jobs from metrics that shouldn't count them by @smudge in #73
- fix(monitor): Obey lock timeout when reporting claimed/unclaimed rows by @smudge in #74
- fix: Include 'claimed' as part of 'claimed_by' scope by @smudge in #75
- perf(monitor): use 'attempts > 0' for better indexability by @smudge in #76
- perf: HOT updates for PostgreSQL, index
locked_atfor everyone else. by @smudge in #77 - perf(worker): avoid sequential scan during worker shutdown by @smudge in #79
- perf: Improve selectivity of pickup query / "claimed" states by @smudge in #80
- perf: remove legacy/unused "delayed_jobs_priority" index by @smudge in #78
- fix: prevent DST +
:localfrom breaking job backoff by @smudge in #81 - fix: Improve
upsert_indexto only drop+rebuild if there is no matching & valid index by @smudge in #83 - test(migration path): ensure that new queries are compatible with old index by @smudge in #84
- perf(monitor): avoid querying the same thing twice, avoid seq scans by @smudge in #82
- build: Ship 2.0.0! by @smudge in #71
Full Changelog: v1.2.1...v2.0.0
v1.2.1
v1.2.0 - `delayed_jobs.name` !!!
What's Changed
This release adds a delayed_jobs.name column that will be populated by default when each job is enqueued. While this column starts as nullable (to ensure zero downtime is required to make this change), it may someday be promoted to a required column (TBD).
The purpose of this column is twofold:
- Improve downstream developers' ability to manually operate the queue without needing to query deep into the
handlercolumn. - Power richer observability & instrumentation tooling (e.g. top N job types in the table, etc)
Full Changelog: v1.1.0...v1.2.0
v1.1.0
v1.0.0 🎉
v0.8.0 - `Priority#to_f`
What's Changed
Full Changelog: v0.7.2...v0.8.0