Segmentation fault at attribute activity

[aarontcopal2]

I am getting a segmentation fault when running hpcrun ./myProgram

Program terminated with signal 11, Segmentation fault.
#0  0x00007f20c3206e96 in cstack_push (q=0x7f20bff2cf68, e=0x20ec8348e5894855) at ../../../../src/lib/prof-lean/stacks.c:223
223	    s_element_t *enext = (s_element_t *) atomic_load(&e->Ad(next));

Stacktrace

#0  0x00007f20c3206e96 in cstack_push (q=0x7f20bff2cf68, e=0x20ec8348e5894855) at ../../../../src/lib/prof-lean/stacks.c:223
#1  0x00007f20c3207015 in bistack_push (s=0x7f20bff2cf68, e=0x7f20b05d97d8) at ../../../../src/lib/prof-lean/bistack.c:85
#2  0x00007f20c32070f5 in bichannel_push (ch=0x7f20bff2cf68, dir=bichannel_direction_forward, e=0x7f20b05d97d8)
    at ../../../../src/lib/prof-lean/bichannel.c:81
#3  0x00007f20c31dbc73 in gpu_activity_t_bichannel_push (c=0x7f20bff2cf68, dir=bichannel_direction_forward, e=0x7f20b05d97d8)
    at ../../../../src/tool/hpcrun/gpu/gpu-activity-channel.c:108
#4  0x00007f20c31dbe10 in gpu_activity_channel_produce (channel=0x7f20bff2cf68, a=0x7f20b1ffcf10)
    at ../../../../src/tool/hpcrun/gpu/gpu-activity-channel.c:157
#5  0x00007f20c31dbf75 in attribute_activity (hc=0x7f20b05d9938, activity=0x7f20b1ffcf10, cct_node=0x7f20bff2cf88)
    at ../../../../src/tool/hpcrun/gpu/gpu-activity-process.c:139
#6  0x00007f20c31dc07a in gpu_memcpy_process (activity=0x7f20b1ffcf10) at ../../../../src/tool/hpcrun/gpu/gpu-activity-process.c:186
#7  0x00007f20c31dc814 in gpu_activity_process (ga=0x7f20b1ffcf10) at ../../../../src/tool/hpcrun/gpu/gpu-activity-process.c:585
#8  0x00007f20c31f4edc in opencl_activity_process (event=0x294f650, user_data=0x293ae90) at ../../../../src/tool/hpcrun/gpu/intel/opencl-api.c:77
#9  0x00007f20c31f4e9a in opencl_buffer_completion_callback (event=0x294f650, event_command_exec_status=0, user_data=0x293ae90)
    at ../../../../src/tool/hpcrun/gpu/intel/opencl-api.c:65
#10 0x00007f20bdd37e4c in ?? () from /lib64/libnvidia-opencl.so.1
#11 0x00007f20bdd3edae in ?? () from /lib64/libnvidia-opencl.so.1
#12 0x00007f20bdd3f650 in ?? () from /lib64/libnvidia-opencl.so.1
#13 0x00007f20bde69b18 in ?? () from /lib64/libnvidia-opencl.so.1
#14 0x00007f20c2f8d35b in monitor_thread_fence2 () at pthread.c:978
#15 0x00007f20c1d6eea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f20c22858cd in clone () from /lib64/libc.so.6

These are the locations of calls to activity and correlation channels.

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-api.c

Line 56 in ab33ace

gpu_activity_channel_consume(gpu_metrics_attribute);

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-api.c

Line 58 in ab33ace

gpu_correlation_channel_produce(correlation_id, &gpu_op_ccts, cpu_submit_time);

(internally calls gpu_correlation_channel_produce)

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-api.c

Line 70 in ab33ace

gpu_monitoring_thread_activities_ready();

hpctoolkit/src/tool/hpcrun/gpu/gpu-activity-process.c

Line 139 in 8b92e78

gpu_activity_channel_produce(channel, activity);

Are the function calls and arguments here correct? I feel the segfault is related to one of these calls being incorrect

[Jokeren]

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-api.c

Line 32 in ab33ace

gpu_correlation_id_map_insert(local_c_id, local_c_id);

It shouldn't be called in the application thread.

hpctoolkit/src/tool/hpcrun/gpu/gpu-activity-process.c

Line 151 in ab33ace

gpu_correlation_id_map_lookup(correlation_id);

The lookup operation and the insert operation should both be called by the background thread only.

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-api.c

Line 31 in ab33ace

__atomic_load(&correlation_id, &local_c_id, __ATOMIC_SEQ_CST); //TODO

Fix the correlation id problem.

And intel.c is still missing.

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-intercept.c

Line 40 in ab33ace

opencl_subscriber_callback(kernel, &correlation_id);

The callback should called before the actual function.

[aarontcopal2]

1. Now the lookup, insert and produce/consume function calls are made by the background threads
2. I have made the correlation_id fetching and increments atomic.
3. intel.c has not been added yet. I will have to use device_notify_map() inside device_initializers.c because I need to dlopen libraries and check if certain functions are present. So I dont see the point of adding a "intel.c" file in sample-sources
4. In Intel's opencl, you can register callbacks for an event only after that event has been submitted. So, I cant change the order of lines here. Here is a link with more details

Sadly, the segmentation faults keep occuring event after making these changes.

[Jokeren]

@aarontcopal2 , opencl_subscriber_callback is not a callback for a event. It's just your own function, you can put it anywhere you like. But if you are sure performance events won't be returned before clSetEventCallback, it is fine to leave the opencl_subscriber_callback unchanged.

[aarontcopal2]

I have made opencl_subscriber_callback as a callback because some of the calls made by it (gpu_correlation_id_map_insert, gpu_activity_channel_consume, gpu_correlation_channel_produce) needed to be made from background threads, right?

[Jokeren]

No, opencl_subscriber_callback is called by the application thread. It is a callback for the function. In contrast, clSetEventCallback sets the (event) callback for buffer completion, which should be executed by the background thread. No sure if you previously had a different understanding.

[Jokeren]

Just noticed two suspicious lines.

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-intercept.c

Line 38 in ab33ace

cl_kernel_callback *kernel_cb = (cl_kernel_callback*) malloc(sizeof(cl_kernel_callback));

You have to fix this later after you fix the current bug.

Frequent malloc winds up wasting time and memory. The correct way to do it is reusing a piece of memory. You can reach out to John to discuss the design of this feature.

https://github.com/HPCToolkit/hpctoolkit/blob/master/src/tool/hpcrun/gpu/nvidia/cupti-activity-translate.c#L652

A cstack_ptr_set is missing for gpu activities in your case. I suppose this is the main reason for your crash.

[aarontcopal2]

hpctoolkit/src/tool/hpcrun/gpu/intel/opencl-intercept.c

Line 46 in 280e8e0

clSetEventCallback(*event, CL_SUBMITTED, &opencl_subscriber_callback, kernel_cb);

Even opencl_subscriber_callback is being called by clSetEventCallback

[Jokeren]

@aarontcopal2 , they are different. Notice the event name is called CL_SUBMITTED. The background thread does not submit any kernels.

[aarontcopal2]

@Jokeren ok, that is a possibility. If thats the case, gpu_correlation_id_map_insert, gpu_activity_channel_consume and gpu_correlation_channel_produce are being called by the main thread. Is this ok or should we try moving these calls to a background thread

[Jokeren]

gpu_correlation_id_map_insert should be called by the background thread.
gpu_correlation_channel_produce should be called by the application thread.
gpu_activity_channel_consume should be called by the application thread.

[aarontcopal2]

@Jokeren your guess was spot-on. Not calling cstack_ptr_set() was the reason of the segfaults.

@jmellorcrummey , should'nt call to cstack_ptr_set() be abstracted? It seems like a low-level detail that should'nt be called between activity_translate() and activity_process() calls

[aarontcopal2]

Let me create another issue for keeping track of missing gpu_metrics. Ill close this issue of segfaults once John replies about cstack_ptr_set