Headers
EJ-FAT uses gRPC messages and UDP packets for its control plane and UDP only for its data plane. E2SAR has 3 pre-defined headers that go immediately after the UDP header: Sync header for Sync messages sent from the sender to the control plane, LoadBalancer (LB) and Reassembly (RE) headers. The latter can appear together (LB+RE) when sent by the data sender and when the load balancer strips off the LB header needed for it, only the RE header and data proceed. As the name suggests, LB header is needed by the Load Balancer and the RE header is needed by the receiver to reassemble the event data.
IP
IP Source = IP address of the Data Source
IP Destination = Sync IP from ReserveLoadBalancerReply
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = Specified in EJFAT URI
UDP Source Port = Any
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L | C | Version | Rsvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| EventSrcId |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| AvgEventRateHz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ UnixTimeNano +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
32b Control Word
16b Magic = 0x4C43 ("LC")
8b Version = 1
8b Rsvd, MBZ
32b EventSrcId
Set this to a unique ID for the Event Source that produced this message
64b EventNumber
Set this to the EventNumber that would be used if an Event Data message were to be sent right now
32b AvgEventRateHz
Set this to the best available estimate of how much the EventNumber increases per second
If no estimate is available, set this field to zero
64b UnixTimeNano
Set this to the current wall clock time of this Event Source. Time is in nanoseconds since
1970-01-01T00:00:00Z (ie. midnight UTC)
If no wall clock timestamp is available, set this field to zero
The C/C++ definition looks as follows:
using EventNum_t = u_int64_t;
using UnixTimeNano_t = u_int64_t;
using EventRate_t = u_int32_t;
struct SyncHdr
{
char preamble[2] {'L', 'C'};
u_int8_t version{synchdrVersion};
u_int8_t rsvd{0};
u_int32_t eventSrcId{0};
EventNum_t eventNumber{0LL};
EventRate_t avgEventRateHz{0};
UnixTimeNano_t unixTimeNano{0LL};
} __attribute__((__packed__));
IP
IP Source = Any
IP Destination = LB Hardware Data Plane IPv4 or IPv6
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = 19522 (0x4C42) (“LB”)
UDP Source Port = Any (typically set to 16 least significant bits of Event Number).
It is important to have entropy in this so that packets are spread across LAGs.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L | B | Version | NextProtocol |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Rsvd | Entropy |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
64b LB Control Word
16b Magic = 0x4C42 (“LB”)
8b Version = 2
8b NextProtocol ID
16b Reserved, MBZ
16b Entropy
64b LB Event Number - this can just be a timestamp
Used to partition the event space into Calendar Epochs
Repeats across multiple packets related to the same event
MUST NOT wrap. Normally filled in by the E2SAR code.
The C/C++ definition looks as follows:
struct LBHdr
{
char preamble[2] {'L', 'B'};
u_int8_t version{lbhdrVersion};
u_int8_t nextProto{rehdrVersion};
u_int16_t rsvd{0};
u_int16_t entropy{0};
EventNum_t eventNum{0L};
} __attribute__((__packed__)); IP
IP Src = Any
IP Dst = LB Hardware Data Plane IPv4 or IPv6
IP Protocol/NextHeader = UDP
UDP
UDP Dst Port
Any value from 16384 to 32767 (inclusive)
NOTE: The UDP destination port must be identical for all segments within the same Event
NOTE: For high rate senders, this value SHOULD provide sufficient entropy for any intermediate
link-aggregation groups to spread the packets uniformly across their links. ie. for each different
event, a different UDP Dst Port value within the range may be selected.
UDP Src Port
Any (typically set to 16 lsbs of Event Number)
NOTE: The UDP destination port must be identical for all segments within the same Event
NOTE: For high rate senders, this value SHOULD provide sufficient entropy for any intermediate
link-aggregation groups to spread the packets uniformly across their links. ie. for each different
event, a different UDP Dst Port value within the range may be selected.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L | B | Version (3) | NextProtocol |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Slot Select | Port Select |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Tick +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Load Balancer Header v3 (located at beginning of UDP payload)
64b LB Ctrl Word
16b Magic = 0x4C42 (“LB”)
8b Version = 3
8b NextProtocol ID
1 = E2SAR Segmentation Header
16b Slot Select
Used to uniformly choose a specific registered receiver (compute node) within the active Calendar Epoch
Repeats across multiple packets related to the same event
MAY be filled in with the 16 lsbs of the Tick field
MAY be filled in with a pseudorandom number, chosen once for each event
NOTE: even when this is pseudorandom, it must be chosen so that it is identical across all senders
when the full event is partitioned across multiple senders (such as when multiple parallel DAQs
observe the same event from multiple perspectives).
For ideal, smooth distribution of events across all registered receivers, this field should include
16-bits of entropy. ie. all possible 16-bit values should occur approximately uniformly over any
window of 65536 events.
16b Port Select
Used used to compute the offset into the per-receiver range of allowed UDP dst ports
MAY be filled with a pseudorandom number, chosen once for each event.
64b Tick
Used to partition the event space into Calendar Epochs
Repeats across multiple packets related to the same event
MUST NOT wrap
C/C++ definition looks as follows:
struct LBHdrV3
{
const char preamble[2] {'L', 'B'};
u_int8_t version{lbhdrVersion3};
u_int8_t nextProto{rehdrVersion};
u_int16_t slotSelect{0};
u_int16_t portSelect{0};
EventNum_t tick{0L};
} __attribute__((__packed__)); IP
IP Source = Any
IP Destination = LB Hardware Data Plane IPv4 or IPv6
IP Protocol/NextHeader = UDP
UDP
UDP Destination Port = 19522 (0x4C42) (“LB”)
UDP Source Port = Any (typically set to 16 least significant bits of Event Number).
It is important to have entropy in this so that packets are spread across LAGs.
LB Header
Stripped off by load balancer and not seen by the receiver
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Rsvd | Data-ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Buffer Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Buffer Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ EventNumber +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4b Version = 1
12b Reserved
16b Data-ID = source identifier
32b Buffer Offset = The byte index of where the data in this packet should be put in the buffer
32b Buffer Length = The size of buffer needed to store this event
64b Event Number = Event number for the application, despite confusing terminology has
**nothing to do** with the Event Number contained in the LB header. These two are treated
independently. Normally filled in by event number provided by the user.
The C/C++ definition looks like this:
struct REHdr
{
u_int8_t preamble[2] {rehdrVersionNibble, 0}; // 4 bit version + reserved
u_int16_t dataId{0}; // source identifier
u_int32_t bufferOffset{0};
u_int32_t bufferLength{0};
EventNum_t eventNum{0};
} __attribute__((__packed__));The role of each field is described in the figure below
