Support for extending database size?

[bclothier]

As I understand it, WhiteDB require that size parameter be provided when creating/loading the database. This doesn't work well in scenarios where the size isn't known in advance and could vary considerably.

If it's not practical to implement extending an open database's memory space, what strategies are available for allocating more memory on demand in a performant manner?

[priitj]

Well, I would agree with you that such scenario is not well supported.

Possible strategies:

• allocate a lot of memory in advance (wastes memory)
• if you have many databases, store them all together and use a field as indicator (still wastes memory, but no more than a single database would)
• split the database into fragments and use some hash-based addressing scheme. Possibly allocate fragments on demand (multiple indexes won't work well, single one could be coerced to work)

[bclothier]

Just for my education, why can't we simply extend? If I understand everything is an offset within a contiguous block of memory, adding new offsets beyond the first block would spill over into the 2nd block that's allocated on the demand. It's acceptable to keep allocating new blocks which can be then linked so that one block handles a set of offsets, right?

I am also considering whether it's acceptable to dump database, resize, load. The only concern is whether this will cause excessive pause between resizes.

[priitj]

AFAIK we don't have enough control over how the shared memory is mapped into the virtual address space. There should be contiguous unused address space available to fit all the segments and future segments for that idea to work. Some platforms do not even let you control where your segment is mapped.

[bclothier]

Ah, gotcha. My scenario actually was for a local database; I do not need it in the shared memory. Would that make a difference?