[Potential Bug] race condition with pack jobs updating root CID

[parkan]

the current implementation has a potential race condition risk because when multiple workers execute pack jobs they race to set the root CID value; if for example the next-to-last job completes after the last it may set the root cid in such a way that a part of the tree will be missing

we need to add a test for this, specifically for the case where the last job is very fast due to having a much smaller hanging segment, or factor out the root cid write into a separate job

there is a transaction to guard against the conflict but the retry may not incorporate directories added by the task that "won" previously if they are on a different sub-tree

need to double-check this

[parkan]

ok the transaction + full resolve seems to address this, I'd like to add a specific test validating this but currently it seems to be ok

[parkan]

it appears that the transaction does indeed guard against an incorrect tree but at the expense of not only every task recalculating it but (worst case) up to o(n^2) recalculations

worth (1) adding a test (2) profiling to examine how long the retries take

[parkan]

the way this appears to work is that you can execute daggen any time as long as a single pack job has completed, this will silently produce a dag that only represents the completed pack jobs