Decorators from deco cannot be used on class methods

Hey,

I know this because, well, it's not implemented.

However since standalone methods are objects in python just as class methods are, what are the main sticking point preventing that from being usable on class methods? Could we get some sort of "what needs to happen" for that to be possible?

I would see a real use for that on class methods. Say you have a DataObject that contains a Gig of data in numpy arrays or something. Maybe that object also defines operations that can be perform on its data. It would be great from a software architecture perspective to keep the object-oriented code and those methods in the class definition, but still be able to parallele process it like you allow it to be done on standalone methods.

Am I am making sense?

[alex-sherman]

I think implementing the functionality is possible, it's just slightly more complicated because while both functions and instance methods are similar, instance methods go through a different life cycle when being called. Every time an instance method is called, an extra __get__ method is called beforehand to "bind" the instance method. This is meant to return a new function now that, when called, will pass the magic self argument do the original instance method. So for deco this just means also passing the self argument to whatever concurrency mechanism is chosen.

The implementation of this would add a decent bit of complexity the library, and it would also allow for people to shoot themselves in the foot very hard! The example you give is actually a great reason to not implement this functionality. The reason being, all concurrent function arguments are pickled when they're passed to the underlying concurrency mechanism. In this case, now all the state that is in the instance, maybe several gigabytes of data, is being sent to a new process which is very time consuming. This process is probably much much slower than just doing all the work in serial.

Additionally deco cannot track mutations objects, it really only knows about the dict type. So implementing this functionality could create the expectation that you can update state within the instance, but those mutations will not be seen outside of the concurrency mechanism.

A brief summary:

• It's possible, but adds complexity
• Its use would be incredibly confusing to users

I definitely get that it's annoying to not be able to use instance methods, the driving goals of deco are to remain simple, and easy to use which, unfortunately, I think instance methods just are not in this context.

Woah quick reply - thanks.

Makes sense, and does tie up into the motivation you highlighted in the paper as to why deco in the first place. I agree with what you're saying about object mutations. However in the use case I have in mind, it's more about calculating derived representations of that object's data (and potentially a bunch of those representations may be required, each requiring heavy computations, hence the idea of using something like deco to not change the code & just parallelize it). So the computations don't mutate the objects. I could probably outsource it to some processing.py where the standalone methods & pass it the data. Or I could see if I could make that work directly with the mp library.

And yeah, you could really shoot yourself in the foot.

Thanks for the clarification (and for the lib too!)"

[alex-sherman]

Yeah just observing the object data does make sense as a use case, but like I said if the object is passed to the worker it will incur this pickling overhead. Like even if you write this sort of thing with the mp library directly (which is ultimately what deco uses), if you pass the large object it will simply be slower.

One solution is to sacrifice the "cleanliness" of the code and reference the objects from outside of the function scope. It's possible to convert the following:

class DataObject:
    @concurrent
    def do_work(self):
        data = self.data
        # and so on

To use global variables which will not be pickled when calling the concurrent functions. This is a pattern I would use with deco and multiprocessing to avoid pickling overhead. The only difference is that I find the deco style slightly cleaner/more declarative.

someObject = DataObject()

@concurrent
def do_work():
    data = someObject.data
    # and so on

TLDR: The big problem is passing large objects to multiprocessing. Pickling large objects is a problem with or without deco, and the solution in both cases is sort of what deco prescribes for general use. (But yeah I totally agree that this is less clean/more difficult to maintain)