Integer/Blob CacheKey Support?

[goodones-mac]

I'm curious why there isn't an option for int cache key support? From what I've inspected your using sqlite as a basic key value store. We've found that string keys of uuids use a lot of storage space relative to their blob equivalents. And if you want to use uuids as the basis of your key system, you could store them directly as 128bit blob objects in the sql database and save a lot of space.

We did a test for our own db recently and it reduced our DB 4x in size by replacing our string UUID keys to int / blob equivalents.

It would be really powerful if we would have an Int option for cacheKeys and/or an optimized version for UUIDs. FileStore / JSONStore could use the string representation, but sqllite and maybe others would use the int / uuid representations directly. This would then bubble up to Boutique

https://stackoverflow.com/questions/11337324/how-to-efficient-insert-and-fetch-uuid-in-core-data/11337522#11337522

[mergesort]

Hey @goodones-mac, this is a really good suggestion! The reason there isn't an option for Int cache keys is pretty simple, it's that you've seemingly spent more time thinking about and researching this problem than I did. 😄

When I started working on Bodega I chose a String as the key because the library was writing files to disk. Afterwards I stuck with it because String is a least common denominator type that is supported across all sorts of backing stores, whether that be the file system, a database, a server, etc.

If you have a good implementation in mind to improves upon what Bodega is doing I'd definitely be interested to integrate that into the library, whether that's using ints or blobs or something else.

---

I have a couple of questions and thoughts, please feel free to tell me if they make sense or not.

• Bodega doesn't only support the SQLiteStorageEngine so we have a constraint of making sure that any solution works in DiskStorageEngine and could be used in any StorageEngine someone decides to build.

• I hadn't made Create Key protocol and deprecate CacheKey when Swift 5.7 is released #9 a priority because I stopped working on Bodega full time last year and the changes I wanted to make required Swift 5.7. Swift 5.7 hadn't yet been released then, and so I think it would be great to pair any changes to how keys work with Create Key protocol and deprecate CacheKey when Swift 5.7 is released #9 to make these changes together with only one deprecation.

• When you say "We did a test for our own db recently and it reduced our DB 4x in size by replacing our string UUID keys to int / blob equivalents" can you quantify how big of a difference that made? If it means going from 4KB to 1KB then I don't think that would be a big enough deal for me to focus energy on, but if it's going from 4MB to 1MB then that's a different story. I'm also wondering did you notice any performance improvements or regressions that should be known ahead of time?

• Can this be done in a way that is backwards compatible? I want to ensure that this change doesn't break any previous databases a user may have. I'm imagining you want to introduce this as an additive change, another way to create keys rather than the only way to do so, but I wanted to check and ensure that my assumption was correct.

---

I'm willing to talk through this or look over any pull requests you may want to push up, it may be just me but it's always a lot easier for me to think through and prioritize work when I can see something tangible rather than talking about problems in the abstract.

Thanks again for the great suggestion!

[goodones-mac]

Hey @goodones-mac, this is a really good suggestion! The reason there isn't an option for Int cache keys is pretty simple, it's that you've seemingly spent more time thinking about and researching this problem than I did. 😄

When I started working on Bodega I chose a String as the key because the library was writing files to disk. Afterwards I stuck with it because String is a least common denominator type that is supported across all sorts of backing stores, whether that be the file system, a database, a server, etc.

If you have a good implementation in mind to improves upon what Bodega is doing I'd definitely be interested to integrate that into the library, whether that's using ints or blobs or something else.

I have a couple of questions and thoughts, please feel free to tell me if they make sense or not.

• Bodega doesn't only support the SQLiteStorageEngine so we have a constraint of making sure that any solution works in DiskStorageEngine and could be used in any StorageEngine someone decides to build.

I think this is fine, since you can fallback to converting the Integers or UUID objects into strings again and back if need be. Also a primary key being a number, int or some other number type of X-bits (which is what UUIDs are, 128 bit numbers) is a pretty well supported thing in the vast majority of DB systems. Integers being a primary key has been the default of how DBs worked for a long time AFAIK. Also in the JSON File case I think it's fine also since there is no real primary key shenanigans and I think they support integers as object keys so i think it works for your file case too.

• I hadn't made Create Key protocol and deprecate CacheKey when Swift 5.7 is released #9 a priority because I stopped working on Bodega full time last year and the changes I wanted to make required Swift 5.7. Swift 5.7 hadn't yet been released then, and so I think it would be great to pair any changes to how keys work with Create Key protocol and deprecate CacheKey when Swift 5.7 is released #9 to make these changes together with only one deprecation.

• When you say "We did a test for our own db recently and it reduced our DB 4x in size by replacing our string UUID keys to int / blob equivalents" can you quantify how big of a difference that made? If it means going from 4KB to 1KB then I don't think that would be a big enough deal for me to focus energy on, but if it's going from 4MB to 1MB then that's a different story. I'm also wondering did you notice any performance improvements or regressions that should be known ahead of time?

So in the SQL-Lite case, there isn't a perf improvement beyond serialization / deserialization reduction, but there is a huge storage benefit which the stack overflow link goes into. Our app is a photo app and works with photokit a lot, and photokit keys are in a UUID/L0/001 B042A890-376B-499B-A66E-0F4C50CCD7E1/L0/001 format and we use it everywhere in our DB. One table that has %80 of our storage space is basically mostly these photoKit IDs, and a person with a 50k photo library ended up with a 55MB DB size. I investigated how to optimize the space usage, recognized that these IDs are hex numbers and replaced that column as a test with an int column with random numbers, and the DB went to 12MB in size.

The way you use string keys probably wont be as dramatic of a space saving mechanism since you use it as a key value store only, but I think it will be pretty significant. If someone is going to do 'random number UUID' as their key type as a default key type because they don't really care about the cachekey type at all, then defaulting to random Int / incremental Int will result in space savings for everyone. 2^64 is a big key space!

• Can this be done in a way that is backwards compatible? I want to ensure that this change doesn't break any previous databases a user may have. I'm imagining you want to introduce this as an additive change, another way to create keys rather than the only way to do so, but I wanted to check and ensure that my assumption was correct.

Yes it would be additive. CacheKey<Int>, CacheKey<UUID>, CacheKey<String>, etc.

I would assume that Int specialized / UUID specialized CacheKeys would go into a separate table or file if types can't be intermixed in the storage engine. And this would be introduced as a new generic specialization so anybody using the current default of string keys probably wont have to do anything. If you change the cache key type it's pretty much a new table although in that case, much like you changing how string cachekeys would be generated would create an equivalent effect.

I'm willing to talk through this or look over any pull requests you may want to push up, it may be just me but it's always a lot easier for me to think through and prioritize work when I can see something tangible rather than talking about problems in the abstract.

Thanks again for the great suggestion!

No problem. Full disclosure, we will probably not adopt this library (so no incoming PRs) because we do need the power of being able to do queries, foreign keys, execute migrations, etc that sqllite & our code currently has, but I'm thinking we might adopt something like MVCS for our app. We already use GRDB & have a repository pattern along with view models, so it might not be too much of a refactor to do.

IMO I would just expand this to supporting UUID, Int and String only and it will give you %99 of the benefit in exchange for a simpler implementation.