Known limitations

[pverscha]

In this issue I'll list some known limitations for both the ShareableMap and ShareableArray that are in most cases a result of limitations posed by the JavaScript programming language (i.e. which are impossible to overcome).

Object references

Objects added to the Map or Array are serialized and deserialized when necessary. This means that if you'll try to retrieve an object that was added to either of these data structures before, a new "identical" object will be returned, but the object references will be different. Contents of the objects are the same, object references are not.

So, be careful when you're relying on this system to, for example, update an object that's retrieved dynamically.

const map = new ShareableMap();
map.set("myKey", []);
// This value will not actually be pushed to the array that was set to `"myKey"` before
// because the array object returned by the Map is not the exact same.
map.get("myKey").push(1);

// Do this instead (and overwrite the array that was set in the Map completely).
const previousArray = map.get("myKey");
previousArray.push(1);
// Overwrite the array in the map completely.
map.set("myKey", previousArray);

ShareableMap is not ordered

In comparison to the Map implementation provided by JavaScript itself, the ShareableMap does not keep track of the ordering of it's items. The ordering is never guaranteed and the order in which items will be returned when iterating over the map is not guaranteed to be deterministic.

[kevinpbaker]

If you're a random person reading this on the internet, and not someone working on the "Unipept" project, then you have most likely ended up as a senior JavaScript developer who has been tasked to build some absurd program that requires large data to be processed across multiple threads while being memory efficient.

Quite the task, especially when trying to do it in a scripting language that wasn't made for such things!

Warning

If you're working with Complex JavaScript Objects and not primitive types, then this library can get you going in the right direction, but read the following carefully before diving into the deep end.

This library saves you from duplicating the data when sending the complex data to the worker(s). You can think of this like "passing the data by reference" (pointers), instead of "passing the data by value" (data copies), which is good. However, if you want to read all the data from the shared array buffer in the worker(s) and compute something based on said data, then it's going to make a copy of the data in each thread as it's deserialized from the bitwise data stored in the buffer, and you're still going to have a larger memory footprint than you might expect.

Note

This is a limitation of the JavaScript language and the engines that run it (like v8) that most likely won't be solved. It's not a problem with this library.

The limitation is that if you want to deserialize the complex data back into javascript objects from the shared memory buffer in order to work with the data, then you're going to be making copies of said data in each thread. Therefore, if you have 50 thousand objects in memory, and you want to read/deserialize the full data in 3 separate threads then you'll have:

• one copy of the serialized data in the shared array buffer
• a copy of the deserialized data in the 1st thread
• a copy of the deserialized data in the 2nd thread
• a copy of the deserialized data in the 3rd thread

That means you'll have around 4x the amount of ram used. Every library that I've looked at can't seem to escape this, because the language doesn't want you todo it. You can't have 1 copy of the complex data in memory that acts like regular java script objects shared across all threads.

If you're experiencing this problem, and searching through hundreds of libraries on GitHub to see how the smartest people around the world are trying to solve something similar, then godspeed!

---

** EDIT 1 **

I updated this comment because I think I did a terrible job at explaining my use case the first go around. The main issue I wanted to warn people against, which is library independent, is that it seems impossible to share regular javascript objects across threads where the underlying data acts as a shared pool of memory, so that you don't have to serialize or deserialize the data as you're working with it.

---

** EDIT 2 **

Eureka moment. The beauty of the data being stored as a map in this library is that you can loop through the keys, deserialize one object reference at a time (avoiding the low-level work approach mentioned by pverscha2 below), process that piece of data, and then free it up for memory collection. By doing this, you should technically be able to avoid the 4x ram usage that I describe above. Only downside is if you need to read all the data at once and do comparisons between objects, then you're still going to run into the memory issue I talked about above!

I think this library might be the gem I was looking for. Thank you for releasing it on GitHub!!!

Hi @kevinpbaker,

Thanks for your valuable feedback on my library. I do have one extra remark though. You can circumvent the issue where you have to keep a copy of the deserialized data in each thread for some complex objects (but it does require some extra low-level work). This library allows you to specify a custom "serializer/deserializer" which makes it possible to create your own custom bitwise representation of a complex data object.

I have initially developed this library for the open-source research project "Unipept" that requires to pass large sets of data between threads. In there, you can find an example that demonstrates exactly what I mean here: https://github.com/unipept/unipept/blob/b3d6d6d47007f0d960a00e21aba82bb9dc08a1f2/src/logic/ontology/peptides/PeptideData.ts

This is a complex JavaScript object that's implemented on top of an ArrayBuffer. You don't need to do completely "serialize/deserialize" it to make it work with SharedCore. If you're interested to learn more, feel free to let me know :)