ofi: Add TX endpoint multiplexing (TxMuxing) to improve transmit throughput

[Copilot]

Adds transmit multiplexing support to the OFI transport, allowing a single SHMEM context to distribute TX operations across multiple OFI endpoints in round-robin order. Each additional endpoint draws from the shared STX pool for hardware-level parallelism while sharing the primary endpoint's counters and CQ — keeping put_quiet/get_wait/fence semantics unchanged.

New environment variable

• SHMEM_OFI_TX_MUX_ENDPOINTS (default: 1 = disabled) — number of TX endpoints per context

shmem_transport_ctx_t (transport_ofi.h)

• Added num_tx_eps, tx_ep_idx, tx_ep_arr[], tx_stx_idxs[]
• Added shmem_transport_ofi_get_tx_ep(ctx): round-robin selector; returns ctx->ep when muxing is off (zero overhead on hot path)
• All 17 TX call sites (fi_write, fi_read, fi_inject_atomic, fi_fetch_atomicmsg, etc.) updated to use the selected endpoint; fi_atomicvalid capability queries retain ctx->ep

shmem_transport_ofi_ctx_init / shmem_transport_ctx_destroy (transport_ofi.c)

• Creates OFI_TX_MUX_ENDPOINTS - 1 additional endpoints; each bound to a shared STX from the pool, and to the shared put_cntr, get_cntr, and CQ
• Endpoint index and STX index written to arrays before bind/enable so partial-init cleanup is safe on any failure
• tx_stx_idxs initialized to -1 sentinel via memset; additional STX references released on destroy independently of ctx->stx_idx
• TX muxing skipped for the default context in single-EP mode (which aliases ctx->ep to target_ep)

Example usage

# Use 4 TX endpoints per context (e.g. with OFI_STX_MAX=4)
SHMEM_OFI_TX_MUX_ENDPOINTS=4 SHMEM_OFI_STX_MAX=4 oshrun -n 16 ./app

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.