Behavior of fence

[anshumang]

Umbrella issue #229
V : not ordered, a fence is required to order
X : not ordered, a fence will not order
O : ordered

The first column in each row specifies the operation and ordering is considered for two consecutive issue of the operation by any given PE to any symmetric object. The second column header same PE is when both the issues are targeted to the same PE. The third column different PE is when the two issues are targeted to two different PEs.

In OpenSHMEM v1.4 :

	same PE	different PE
memory store	V	X
p/put/iput	V	X
put_nbi	V	X
non-fetch AMOs	V	X
memory load	O	O
g/get/iget	O	O
get_nbi	X	X
fetch AMOs	V	X

In proposal :

	same PE	different PE
memory store	V	X
p/put/iput	V	X
put_nbi	V	X
non-fetch AMOs	V	X
memory load	V	V
g/get/iget	V	V
get_nbi	V	X
fetch AMOs	O	O

[jdinan]

Can you add some prose to help us understand what's posted here? I assume that same/different PE means the given operation class when two are issued by the same PE with an intervening fence, versus when two are issued by different PEs? Where and when is the fence performed in the latter case? Also, how does O relate to fence?

[anshumang]

@jdinan I have added text to clarify. Is the table still hard to understand?

[jdinan]

Do operations target the same or different memory locations? Also, isn't ordering of load instructions architecture dependent?

[anshumang]

@jdinan I have updated the text to indicate that the target location can be different. Let me know if it is not clear.

Also, isn't ordering of load instructions architecture dependent?

But should shmem_fence not enforce ordering of memory loads on memory obtained by shmem_ptr?

[jdinan]

In that case, should memory load be V in OpenSHMEM 1.4? If not, O seems like it would be architecture dependent (since they are not OpenSHMEM operations, OpenSHMEM cannot guarantee ordering by default).

[anshumang]

I agree that memory load should be V but it is missing in OpenSHMEM v1.4 spec -
Assures ordering of delivery of Put, AMO, memory store, and nonblocking Put routines to symmetric data objects.

In absence of memory load above, I chose O with this interpretation of v1.4 spec - ordered because shmem_fence is not required.

[jdinan]

IIRC, loads are currently ordered only by shmem_wait and possibly shmem_barrier/sync. My gut feeling is that shmem_fence should not guarantee load ordering since it doesn't ensure that updates are present and visible in memory.

[naveen-rn]

I suppose this is not for OpenSHMEM-1.5.