Slot builder performance

[benbierens]

Seen in devnet.
Slots of 128MB are taking up to 20 seconds to be built by storage nodes.
All of that time sits between
Building slot tree builder.nim line 248
and
Storing slot tree builder.nim line 255

Looking down the call chain:

• blocks are loaded: await self.store.getBlock(self.manifest.treeCid, blkIdx)
• block trees are created tree =? SomeTree.digestTree(blk.data, self.cellSize.int) this is a blocking operation
• slot tree is created SomeTree.init(cellHashes) this is a blocking operation

We can't see from the current logging which of these steps is taking most of the time. The hashing function used here is Poseidon2, so it's not exactly fast. The data is hashed in small 2kb cells. But since there is quite a bit, hashing the tree in a blocking manner appears to still be disruptive to the total node process.

Is there a way to move the entire tree-building operation to the threadpool? Is there a clever way to model this process as a streaming operation, to avoid working-memory constraints for very large slots?