Segfaults | Memory corruption errors

[tjad]

I'm receiving some strange errors when using this gem in conjunction with eventmachine.

I'm rewriting values into a single NDArray instance (reused). Up to 140 writes per second.

Randomly my event loop crashed from mxnet segfault errors. I'm going to push a list of errors in this thread.

These errors appear regardless of my context, so for simplicity, assume these are in CPU context.

1.)
malloc_consolidate(): invalid chunk size

2.) In this error, there are sometimes unicode characters present for attributes as opposed to blank

ruby/2.6.0/bundler/gems/mxnet.rb-edc85f60baa5/lib/mxnet/ndarray/operation_delegator.rb:91:in `imperative_invoke': MXNetError: Cannot find argument '', Possible Arguments: (MXNet::Error)
----------------
scalar : double, optional, default=0
    The scalar value for assignment.
begin : Shape(tuple), required
    starting indices for the slice operation, supports negative indices.
end : Shape(tuple), required
    ending indices for the slice operation, supports negative indices.
step : Shape(tuple), optional, default=[]
    step for the slice operation, supports negative values.
, in operator _slice_assign_scalar(name="", scalar="267.55", step="[1, 1]", end="[181, 1]", ="[180, 0]")

3.) basic_string::_M_construct null not valid

[tjad]

Initially was using 1.5.1 , now using 1.3.1 same results

[tjad]

From what I can see, sometimes the value is corrupted, and sometimes the key is corrupted through imperative_invoke.

It seems that key/values are arriving at imperative_invoke correctly, and hence being corrupted in the C binding.

Did I mention, this happens at some random later stage after setting values several times.

[tjad]

By these images you will see that 590 writes to this array using the same method, one by one in an iterative manner.

Input:

Error:

** Imperative invoke in operation delegator ** :

mxnet.rb/lib/mxnet/ndarray/operation_delegator.rb

Line 99 in edc85f6

return LibMXNet.imperative_invoke(#{handle}, ndargs, keys, vals, out)

[tjad]

It is very likely that this is caused by Ruby's garbage collector. This happens as NDArray has non-blocking manipulations, hence the method can exit flagging its variables for garbage collection before mxnet has finished the execution of the manipulations.

One way to deal with this is to make a copy of the string buffers as opposed to passing the internal ruby string buffers around. This however incurs the unwanted performance hit. We will need to stop marking the memory for garbage collection in order to gain that performance.

https://github.com/mrkn/mxnet.rb/blob/master/ext/mxnet/libmxnet.c#L158

@mrkn Please confirm

[tjad]

Making a copy of the memory wont work without introducing manual garbage collection anyway.

Perhaps creating a pre garbage collector hook that awaits all write ops on NDArray instances

[tjad]

Premature garbage collection is definitely the cause of memory corruption problems. I've added a wait_on_read as a work around after all my writes. This is obviously not great for performance. I'm looking into placing NDArray tracking for the GC so that it can efficiently await all writes of the NDArrays before collecting garbage.

[mrkn]

@tjad Could you show your code to reproduce this problem?

I looked at your commit Yobisense@7a00289.
Calling rb_gc_mark after rb_str_tmp_new is meaningless because rb_gc_mark should be called during the marking process of garbage collection.

Could you show your code to reproduce this problem?

I suspected that string objects allocated at this line may be collected by GC.
Did you specify keys by symbols in your code?

[mrkn]

Both keys_str and values_str needn't be marked explicitly because they are in the stack.
We need to add an array object to keep string objects created by rb_sym_to_s.

[tjad]

@mrkn It's quite involved actually. You are correct in that we shouldn't need to mark the objects as they are on the stack, however, they are passed to MXNet in async, hence method exits and stack is popped including its variables, GC will not mark these and collect if there are no other strong references. This happens when the MXNet engine is not processing tasks/instructions fast enough, and the ruby application is creating objects frequently, causing frequent minor garbage collections. The memory passed to imperative_invoke may be either
a.) Moved (GC is allowed to)
b.) Garbage collected.

To truly solve this problem, we'd have to make use of the MXNet callbacks for the best reliablity. We would have to keep an array of strong references to all those memory being passed to mxnet and clear the references from the array when the operations are complete. At least this is what I'm seeing.

This can be done efficiently by means of a reference pool that can expand as required

Manually marking anything in the stack will only help to some extent. When the GC passes 2 cycles even after manual marking the memory will be garbage collected.

[tjad]

I will try put together a test case. Hopefully the intuition above is sound though to get thinking going.

I think generally ruby applications keep data in memory (hence have a strong reference which stops the GC from clearing that data). In my application specifically, I'm running realtime data which is not stored in the heap whatsoever, so that memory may be cleared when GC runs as they are stack values without strong references. I am writing those values directly to NDArray, using slice assignment. It's A LOT of operations that likely fill up MXNet's task queue, and then whilst doing heavy operations (calculations), the assignments keep piling up and don't get processed quickly enough.

There is some change I can make in my application to avoid some of this, but ultimately it's a problem waiting to happen. It would be great to resolve this elegantly through a manner that we generally don't have to think about memory management.

[tjad]

@mrkn I tried marking keys_str and vals_str as I ran out of ideas, and just started marking everything. Ideally, params_keys and params_vals would be marked after all elements are set, to make use of GC's recursive marking behaviour. Of course, this doesn't help as per above - too many GC cycles before mxnet can process all operations.

[tjad]

As an aside (not sure how relevant this is). When using jemalloc, I never received an error like in 2.)
I only receive error 3.) Dangling references

When using standard malloc, I get 2.) corrupt memory

I have not been able to successfully run TCMalloc with google perftools.

Hence, currently I am using jemalloc. I noticed that in mxnet 1.6.x jemalloc was removed as a default and not recommended in default use. (for whatever reason)

[tjad]

And yes @mrkn , my keys are symbols.