DARTS scheduler assert failure

[Muxas]

Steps to reproduce

I am trying to use DARTS scheduler from the latest master branch (commit 4131e05d441f6aa3004632c61e982c63f2496cb9 of Gitlab) and get the following error:

/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1aa29a)[0x15548dfc629a]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1aacb6)[0x15548dfc6cb6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_push_task+0x4b)[0x15548dfc19f6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pump_to+0x67)[0x15548dfc1c96]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1e7234)[0x15548e003234]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_send_can_push_to_parents+0x191)[0x15548dfc3f14]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1aade3)[0x15548dfc6de3]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pump_to+0x45)[0x15548dfc1c74]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pump_downstream+0xc5)[0x15548dfc1da9]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1ab662)[0x15548dfc7662]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_can_push+0x111)[0x15548dfc3a15]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1b30c4)[0x15548dfcf0c4]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1b3232)[0x15548dfcf232]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x3a)[0x15548dfc1b64]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1be3f6)[0x15548dfda3f6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x3a)[0x15548dfc1b64]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x19fcd6)[0x15548dfbbcd6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x3a)[0x15548dfc1b64]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_tree_pop_task+0x55)[0x15548dfc1b0c]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0xbc34b)[0x15548ded834b]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0xfefaa)[0x15548df1afaa]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(_starpu_cpu_driver_run_once+0x42a)[0x15548e00b8ce]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x1f0207)[0x15548e00c207]
/lib64/libpthread.so.0(+0x7ea5)[0x15555511dea5]
/lib64/libc.so.6(clone+0x6d)[0x155554535b0d]
python3: ../../src/sched_policies/component_fifo.c:89: fifo_push_local_task: Assertion `0 && "starpu_sched_component_can_execute_task(component,task)"' failed.

Full backtrace and config.log are here

At the same time, other schedulers, e.g, DMDASD, work without a problem

[Muxas]

I have broken this problem down to a simple example:

#include <starpu.h>
#include <cuda.h>

void copy_func(void *buffers[], void *cl_args)
{
    float *x = (float *)STARPU_VARIABLE_GET_PTR(buffers[0]);
    float *y = (float *)STARPU_VARIABLE_GET_PTR(buffers[1]);
    cudaStream_t stream = starpu_cuda_get_local_stream();
    cudaMemcpyAsync(y, x, sizeof(float), cudaMemcpyDeviceToDevice, stream);
    printf("copy_func\n");
}

struct starpu_codelet copy_codelet =
{
    .cuda_funcs = {copy_func},
    .cuda_flags = {STARPU_CUDA_ASYNC},
    .modes = {STARPU_R, STARPU_W},
    .nbuffers = 2
};


int main(int argc, char **argv)
{
    float *ptr;
    starpu_data_handle_t x_handle, y_handle;
    starpu_init(NULL);
    starpu_variable_data_register(&x_handle, -1, 0, sizeof(float));
    starpu_variable_data_register(&y_handle, -1, 0, sizeof(float));
    starpu_data_acquire(x_handle, STARPU_W);
    ptr = (float *)starpu_data_get_local_ptr(x_handle);
    *ptr = 1.0;
    starpu_data_release(x_handle);
    starpu_task_insert(&copy_codelet, STARPU_R, x_handle, STARPU_W, y_handle, 0);
    starpu_task_wait_for_all();
    starpu_data_unregister(x_handle);
    starpu_shutdown();
}

Creating X on CPU and using it on GPU to copy from X into Y leads to assert errors of the DARTS scheduler. Running this example sometimes goes without errors and sometimes leads the described problem:

STARPU_NCPU=1 STARPU_NCUDA=1 STARPU_SCHED=darts ./a.out
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x11c46a)[0x15555501446a]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_push_task+0x2a)[0x15555501166a]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pump_to+0x79)[0x155555011a29]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x13b1d9)[0x1555550331d9]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_send_can_push_to_parents+0x4b)[0x155555012cbb]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x11b99b)[0x15555501399b]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x1555550117c6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x129607)[0x155555021607]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x1555550117c6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x116838)[0x15555500e838]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x1555550117c6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x86e95)[0x155554f7ee95]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0xb76ee)[0x155554faf6ee]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(_starpu_cpu_driver_run_once+0x8c)[0x15555503a4ec]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x142c35)[0x15555503ac35]
/lib64/libpthread.so.0(+0x7ea5)[0x155554ce3ea5]
/lib64/libc.so.6(clone+0x6d)[0x1555544ecb0d]
a.out: ../../src/sched_policies/component_fifo.c:89: fifo_push_local_task: Assertion `0 && "starpu_sched_component_can_execute_task(component,task)"' failed.
Aborted

[Muxas]

Even simpler example when DARTS fails with asserts:

#include <starpu.h>
#include <cuda.h>

void clear_func(void *buffers[], void *cl_args)
{
    float *x = (float *)STARPU_VARIABLE_GET_PTR(buffers[0]);
    cudaStream_t stream = starpu_cuda_get_local_stream();
    cudaMemsetAsync(x, 0, sizeof(float), stream);
    printf("clear_func\n");
}

struct starpu_codelet clear_codelet =
{
    .cuda_funcs = {clear_func},
    .cuda_flags = {STARPU_CUDA_ASYNC},
    .modes = {STARPU_W},
    .nbuffers = 1
};


int main(int argc, char **argv)
{
    float *ptr;
    starpu_data_handle_t x_handle[10];
    starpu_init(NULL);
    for(int i = 0; i < 10; ++i)
    {
        starpu_variable_data_register(&x_handle[i], -1, 0, sizeof(float));
    }
    for(int i = 0; i < 10; ++i)
    {
        starpu_task_insert(&clear_codelet, STARPU_W, x_handle[i], 0);
    }
    starpu_task_wait_for_all();
    for(int i = 0; i < 10; ++i)
    {
        starpu_data_unregister(x_handle[i]);
    }
    starpu_shutdown();
}

[sthibaul]

@MaximeGonthier: could you have a look? The problem is not calling starpu_worker_can_execute_task_first_impl to check that the task can be run by the PU you are aiming. That'll have to be done when picking up a random task at the beginning, when selecting a "good data", and when pushing all the "free tasks": only those than can actually be executed there should be pushed

[sthibaul]

(@Muxas' testcases can be really interesting for DARTS, and most starpu testsuite failures and currently due to this issue)

[MaximeGonthier]

Thank you for the examples. I'll work on it as soon as possible. I'll keep you updated

[MaximeGonthier]

@Muxas The Assertion 0 && "starpu_sched_component_can_execute_task(component,task)"'` error has been resolved for the DARTS scheduler on the master branch.
Please let me know if this issue arise again or if any other issue with DARTS is found.

[Muxas]

Trying latest 882aba682cec925bf6bd226c210641bd80b0795d commit shows there is still a problem. I tried this example:

#include <starpu.h>
#include <cuda.h>

void clear_func(void *buffers[], void *cl_args)
{
    float *x = (float *)STARPU_VARIABLE_GET_PTR(buffers[0]);
    cudaStream_t stream = starpu_cuda_get_local_stream();
    cudaMemsetAsync(x, 0, sizeof(float), stream);
    printf("clear_func\n");
}

struct starpu_codelet clear_codelet =
{
    .cuda_funcs = {clear_func},
    .cuda_flags = {STARPU_CUDA_ASYNC},
    .modes = {STARPU_W},
    .nbuffers = 1
};


int main(int argc, char **argv)
{
    float *ptr;
    starpu_data_handle_t x_handle[10];
    starpu_init(NULL);
    for(int i = 0; i < 10; ++i)
    {
        starpu_variable_data_register(&x_handle[i], -1, 0, sizeof(float));
    }
    for(int i = 0; i < 10; ++i)
    {
        starpu_task_insert(&clear_codelet, STARPU_W, x_handle[i], 0);
    }
    starpu_task_wait_for_all();
    for(int i = 0; i < 10; ++i)
    {
        starpu_data_unregister(x_handle[i]);
    }
    starpu_shutdown();
}

Here is a config.log

The error itself:

/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0xbbbd1)[0x155554fb3bd1]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_data_expected_transfer_time+0x83)[0x155554f774e3]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x137453)[0x15555502f453]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x13d323)[0x155555035323]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pump_to+0x63)[0x155555012023]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x13c3c5)[0x1555550343c5]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_send_can_push_to_parents+0x4b)[0x1555550132cb]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x11bfab)[0x155555013fab]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x155555011dd6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x129c17)[0x155555021c17]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x155555011dd6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x116e48)[0x15555500ee48]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(starpu_sched_component_pull_task+0x16)[0x155555011dd6]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x87125)[0x155554f7f125]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0xb797e)[0x155554faf97e]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(_starpu_cpu_driver_run_once+0x8c)[0x15555503a89c]
/trinity/home/al.mikhalev/Install/starpu-1.5-a100/lib/libstarpu-1.4.so.1(+0x142fe5)[0x15555503afe5]
/lib64/libpthread.so.0(+0x7ea5)[0x155554ce3ea5]
/lib64/libc.so.6(clone+0x6d)[0x1555544ecb0d]

[starpu][_starpu_select_src_node][assert failure] The data for the handle 0xaa9d50 is requested, but the handle does not have a valid value. Perhaps some initialization task is missing?

a2.out: ../../src/datawizard/coherency.c:69: _starpu_select_src_node: Assertion `0 && "src_node_mask != 0"' failed.

Full backtrace:

(gdb) bt full
Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
Python Exception <type 'exceptions.ImportError'> No module named gdb.frames: 
#0  0x0000155554424387 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000155554425a78 in abort () from /lib64/libc.so.6
No symbol table info available.
Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
#2  0x000015555441d1a6 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
#3  0x000015555441d252 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
#4  0x0000155554fb3c21 in _starpu_select_src_node (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
handle=handle@entry=0xaa9d50, destination=destination@entry=0)
    at ../../src/datawizard/coherency.c:69
        src_node = -1
        i = <optimized out>
        nnodes = 2
        node = <optimized out>
        size = 4
        cost = inf
        src_node_mask = <optimized out>
        __func__ = "_starpu_select_src_node"
        __PRETTY_FUNCTION__ = "_starpu_select_src_node"
        i_ram = <optimized out>
        i_gpu = <optimized out>
        i_disk = <optimized out>
#5  0x0000155554f774e3 in starpu_data_expected_transfer_time (mode=STARPU_R, memory_node=0, handle=0xaa9d50)
    at ../../src/core/perfmodel/perfmodel.c:458
        size = 4
        duration = 0
        src_node = <optimized out>
        size = <optimized out>
        duration = <optimized out>
        src_node = <optimized out>
#6  starpu_data_expected_transfer_time (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
handle=0xaa9d50, memory_node=memory_node@entry=0, mode=mode@entry=STARPU_R)
    at ../../src/core/perfmodel/perfmodel.c:433
        size = <optimized out>
        duration = <optimized out>
        src_node = <optimized out>
#7  0x000015555502f453 in _starpu_darts_scheduling_3D_matrix (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
main_task_list=main_task_list@entry=0xa91ba0, 
--Type <RET> for more, q to quit, c to continue without paging--
    current_gpu=current_gpu@entry=0, g=0xa91bf0, current_worker_id=<optimized out>) at ../../src/sched_policies/darts.c:1889
        i = <optimized out>
        e = 0xaa2570
        remaining_expected_length_max = 0
        best_1_from_free_task = 0x0
        temp_best_1_from_free_task = <optimized out>
        number_free_task_max = 0
        temp_number_free_task_max = <optimized out>
        number_1_from_free_task_max = 0
        temp_number_1_from_free_task_max = <optimized out>
        priority_max = -2147483648
        temp_priority_max = <optimized out>
        transfer_time_min = 1.7976931348623157e+308
        temp_transfer_time_min = <optimized out>
        ratio_transfertime_freetask_min = 1.7976931348623157e+308
        temp_length_free_tasks_max = <optimized out>
        handle_popped = 0x0
        hud = <optimized out>
        data_chosen_index = 0
        __PRETTY_FUNCTION__ = "_starpu_darts_scheduling_3D_matrix"
        __func__ = "_starpu_darts_scheduling_3D_matrix"
        data_not_available = <optimized out>
        data_available = true
        choose_best_data_threshold = 2147483647
        i = <optimized out>
#8  0x0000155555035323 in get_task_to_return_pull_task_darts (current_worker_id=<optimized out>, l=0xa91ba0, current_gpu=0)
    at ../../src/sched_policies/darts.c:2531
        task = <optimized out>
        __func__ = "get_task_to_return_pull_task_darts"
        __PRETTY_FUNCTION__ = "get_task_to_return_pull_task_darts"
        task = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        task = <optimized out>
        i = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        hud = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        task = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        i = <optimized out>
        hud = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
        p_ret = <optimized out>
        __ptrs = {<optimized out> <repeats 32 times>}
        __n = <optimized out>
#9  darts_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=<optimized out>, to=<optimized out>) at ../../src/sched_policies/darts.c:2685
        data = <optimized out>
        __func__ = "darts_pull_task"
        current_gpu = 0
        task = <optimized out>
#10 0x0000155555012023 in starpu_sched_component_pump_to (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=component@entry=0xa91a90, child=child@entry=0xa91d80, 
    success=success@entry=0x15550c87b40c) at ../../src/sched_policies/component_sched.c:411
        ret = 0
        task = <optimized out>
#11 0x00001555550343c5 in darts_can_push (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=0xa91a90, to=0xa91d80) at ../../src/sched_policies/darts.c:3113
        didwork = 0
--Type <RET> for more, q to quit, c to continue without paging--
        task = <optimized out>
        __func__ = "darts_can_push"
#12 0x00001555550132cb in starpu_sched_component_send_can_push_to_parents (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=component@entry=0xa91d80)
    at ../../src/sched_policies/component_sched.c:685
        __PRETTY_FUNCTION__ = "starpu_sched_component_send_can_push_to_parents"
        i = 0
        ret = 0
#13 0x0000155555013fab in fifo_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=0xa91d80, to=<optimized out>)
    at ../../src/sched_policies/component_fifo.c:179
        __PRETTY_FUNCTION__ = "fifo_pull_task"
        data = 0xa91e90
        queue = 0xa91e90
        mutex = 0xa91ee0
        now = 775762.92599999998
        __func__ = "fifo_pull_task"
        task = <optimized out>
#14 0x0000155555011dd6 in starpu_sched_component_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
from=0xa91d80, to=to@entry=0xa95a90)
    at ../../src/sched_policies/component_sched.c:391
        task = <optimized out>
#15 0x0000155555021c17 in best_implementation_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=0xa95a90, from=<optimized out>)
    at ../../src/sched_policies/component_best_implementation.c:108
        task = 0x0
        i = 0
#16 0x0000155555011dd6 in starpu_sched_component_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
from=0xa95a90, to=to@entry=0xa957b0)
    at ../../src/sched_policies/component_sched.c:391
        task = <optimized out>
#17 0x000015555500ee48 in simple_worker_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
component=0xa957b0, to=<optimized out>)
    at ../../src/sched_policies/component_worker.c:464
        now = 775762.67500000005
        workerid = 1
        worker = <optimized out>
        data = 0xa958c0
        list = 0xa95a20
        task = 0x0
        i = 0
        n_tries = 2
--Type <RET> for more, q to quit, c to continue without paging--
        __func__ = "simple_worker_pull_task"
#18 0x0000155555011dd6 in starpu_sched_component_pull_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
from=0xa957b0, to=0x0)
    at ../../src/sched_policies/component_sched.c:391
        task = <optimized out>
#19 0x0000155554f7f125 in _starpu_pop_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
worker=worker@entry=0x1555552bd160 <_starpu_config+5952>)
    at ../../src/core/sched_policy.c:1066
        sched_ctx = 0x1555552fde08 <_starpu_config+271336>
        task = 0x0
        worker_id = <optimized out>
        node = <optimized out>
        profiling = 0
        pop_start_time = {tv_sec = 1, tv_nsec = 4294967295}
        pick = <optimized out>
        i = <optimized out>
        nbuffers = <optimized out>
#20 0x0000155554faf97e in _starpu_get_worker_task (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
worker=worker@entry=0x1555552bd160 <_starpu_config+5952>, 
    workerid=workerid@entry=1, memnode=memnode@entry=0) at ../../src/drivers/driver_common/driver_common.c:431
        task = <optimized out>
        keep_awake = <optimized out>
        __func__ = "_starpu_get_worker_task"
#21 0x000015555503a89c in _starpu_cpu_driver_run_once (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
cpu_worker=cpu_worker@entry=0x1555552bd160 <_starpu_config+5952>)
    at ../../src/drivers/cpu/driver_cpu.c:617
        memnode = 0
        workerid = 1
        pi = {conf = 0x0, event_type = starpu_prof_tool_event_driver_init_end, starpu_version = {1, 4, 99}, 
          thread_id = 210409216, worker_id = 1, task_name = 0x0, model_name = 0x0, device_number = 0, 
          driver_type = starpu_prof_tool_driver_cpu, memnode = 4294967295, bytes_to_transfer = 0, bytes_transfered = 0, 
          fun_ptr = 0x0}
        res = <optimized out>
        j = <optimized out>
        task = 0x0
        pending_task = 0x0
        rank = 0
        __func__ = "_starpu_cpu_driver_run_once"
        continuation_wake_up = <optimized out>
        __PRETTY_FUNCTION__ = "_starpu_cpu_driver_run_once"
--Type <RET> for more, q to quit, c to continue without paging--
#22 0x000015555503afe5 in _starpu_cpu_worker (Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
arg=0x1555552bd160 <_starpu_config+5952>) at ../../src/drivers/cpu/driver_cpu.c:729
        worker = 0x1555552bd160 <_starpu_config+5952>
        pi = {conf = 0x0, event_type = starpu_prof_tool_event_start_transfer, starpu_version = {1, 4, 99}, 
          thread_id = 210409216, worker_id = 1, task_name = 0x0, model_name = 0x0, device_number = 1, 
          driver_type = starpu_prof_tool_driver_cpu, memnode = 0, bytes_to_transfer = 0, bytes_transfered = 0, fun_ptr = 0x0}
#23 0x0000155554ce3ea5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
Python Exception <type 'exceptions.NameError'> Installation error: gdb._execute_unwinders function is missing: 
#24 0x00001555544ecb0d in clone () from /lib64/libc.so.6
No symbol table info available.

[MaximeGonthier]

Ah, thanks! I'm working on it.

[sthibaul]

@MaximeGonthier does DARTS pay attention to STARPU_TASK_GET_MODE(task, i) ? tasks only need to load data when (STARPU_TASK_GET_MODE(task, i) & STARPU_R) != 0 . We indeed probably never tried DARTS on a task with only STARPU_W.

[sthibaul]

@MaximeGonthier probably the case that DARTS was never tried against was task_insert(t1, A, STARPU_W); task_insert(t2, A, STARPU_R); task_insert(t3, A, STARPU_W), in which case the value computed by t1 is indeed dropped once t2 is over. In other words, when DARTS receives a task that only writes to a data, it should mostly forget about this data.

[MaximeGonthier]

DARTS uses STARPU_TASK_GET_MODE only to ignore data of type STARPU_SCRATCH and STARPU_REDUX, nothing else.
But this should not cause any issue since DARTS does not directly load data it just returns a task. Maybe it's when it reads the data that it causes a crash?

In other words, when DARTS receives a task that only writes to a data, it should mostly forget about this data.

Yes ok I see I'll work on that, thanks

[sthibaul]

But this should not cause any issue since DARTS does not directly load data it just returns a task. Maybe it's when it reads the data that it causes a crash?

See the backtrace: DARTS is calling starpu_data_expected_transfer_time, but when the data is getting discarded by a STARPU_W access, there is no data to transfer, so calling that function does not make sense. But another way, the presence of the hardcoded STARPU_R in DARTS is bogus, it has to use the task's access mode.

[MaximeGonthier]

Yes I figured that out just now that starpu_data_expected_transfer_time is the culprit in that case :)

the presence of the hardcoded STARPU_R in DARTS is bogus

Yes I agree. Do you think I should check STARPU_TASK_GET_MODE(task, i) before each time a data is read or only for functions like starpu_data_expected_transfer_time that are suppose to give me an output that cannot exist if it's only a W

[sthibaul]

Yes I agree. Do you think I should check STARPU_TASK_GET_MODE(task, i) before each time a data is read

It depends what you are doing with the data. If e.g. you record where a piece of data will be (to tend to put tasks that will read it there), you'll still want to see that even if the mode is STARPU_W.

[MaximeGonthier]

@Muxas the issue is fixed in the last commit from master (eedd54ac). The simple example presented above is now working
Thanks