Enabling OOC causes "handle is not initialized" error

[Muxas]

Hi!

I am trying to get OOC working and keep getting an error

[starpu][_starpu_data_check_initialized][assert failure] handle 0x55555db36e50 is not initialized while trying to read it

I enable OOC support by simply providing 3 environment variables:

export STARPU_DISK_SWAP=/tmp
export STARPU_DISK_SWAP_BACKEND=unistd
export STARPU_DISK_SWAP_SIZE=200

Without these exports, everything works correctly without any errors.

All handles are initialized without any pointer to make StarPU in charge of all allocations and deallocations. Actual data is provided to handles by the main thread with a help of acquire-release mechanism. Maybe it will help finding the problem.

Backtrace

Backtrace is attached

backtrace.txt

Configuration

I am using StarPU-1.4.7 (tagged starpu-1.4.7)

Configuration result

Config log is attached

config.log

Update 1

Eliminating starpu_data_wont_use() everywhere in the code resolved the issue. Half year ago I already posted a bug, related to this function #56

[Muxas]

Update 2

After digging in the StarPU code and commits, I found, that the issue was resolved by commit cd726c29aa48f55da0ef188c49f4132010748492.

And now I am reading this code

starpu/src/datawizard/user_interactions.c

Lines 691 to 706 in 3003abf

	if (handle->home_node != -1)
	__starpu_prefetch_data_on_node_with_mode(handle, handle->home_node, 1, STARPU_R, STARPU_IDLEFETCH, STARPU_DEFAULT_PRIO);
	else
	{
	if (handle->ooc)
	{
	/* Try to push it to some disk */
	unsigned i;
	unsigned nnodes = starpu_memory_nodes_get_count();
	for (i = 0; i < nnodes; i++)
	{
	if (starpu_node_get_kind(i) == STARPU_DISK_RAM)
	__starpu_prefetch_data_on_node_with_mode(handle, i, 1, STARPU_R, STARPU_IDLEFETCH, STARPU_DEFAULT_PRIO);
	}
	}
	}

All my tensors are processed in 2 stages: producer stage and consumer stage. After the producer stage each tensor is marked by starpu_data_wont_use(). After the consumer stage, all tensors are marked by starpu_data_invalidate_submit() to be reused in a new producer-consumer cycle.

My code does not allocate memory by itself, which means there is no home node. Therefore, starpu_data_wont_use() effectively does nothing in my case. Instead of offloading data from GPU to CPU in the background (while bus is idle), nothing really happens.

As soon as I enable OOC, my code immediately slows down by a factor of infinity. Code, that was running 30 seconds without OOC does not finish in 20 minutes with OOC enabled. As I see in the above mentioned code, every tensor is pushed into the slow disk. As far as I understand it, starpu_data_wont_use() does not take into account amount of available memory and the need to offload the data, but orders to offload it. There is no way to cancel this order. And my code waits until all the tensors are pushed through the bus. Am I correct?