# Integration with EJFAT dataplane and control plane ## Overview E2SAR uses a combination of UDP packets and gRPC+TLS to communicate with loadbalancer control plane and dataplane. The process of using the load balancer begins with (1) reserving a load balancer using an out-of-band information like the ejfat URI of the Control Plane agent with embedded `admin token`. This can be done by Workflow Management System (WMS) or its proxy. Once the load balancer is reserved and the details of the reservation communicated out of band to the worker nodes via another EJFAT URI, they can register themselves with the control plane (2). Alternatively the registration can be peformed by WMS or its proxy assuming the details of the workers are known to it. All senders are required to be registered with the Control Plane by using AddSenders gRPC call (7). Otherwise their traffic will be rejected at the Load Balancer. When a sender is no longer needed, it can be unregistered using RemoveSenders call [8]. Either call can be invoked by the sender node or by the WMS. At this point data can begin traversing the dataplane in segments (3a and 3b). Periodic sync messages (4) are required to be sent to the Control Plane by the sender. Similarly the receiving nodes or WMS on their behalf must periodically use SendState gRPC call (5) to update queue occupancy information in the control plane. After the workflow completes or if worker software fails, the workers can be deregistered (6) - themselves or via WMS. The load balancer can be freed (9). In addition at any time someone with admin token can request repeat reservation reply via GetLoadBalancer call (X), get detailed information about load balancer and its worker node clients via LoadBalancerStatus (Y) and get an overview of all provisioned load balancers and their associated worker node clients via Overview call (Z). ![E2SAR and Control Plane Messaging](fig/e2sar-and-cp.png) ## EJFAT_URI formats EJFAT defines its own URI format to convey necessary information to different actors in different phases of the workflow. The general format of the URI is: ``` ejfat[s]://[@]:/[lb/][?[data=]>][&sync=:][&sessionid=]]. ``` Where: - `ejfat` or `ejfats` is the schema name, with `ejfats` requiring the use of TLS when speaking gRPC to the Control Plane (CP) - `token` is either admin, instance or session token, depending on the context (discussed below) needed to communicate over gRPC (it is not used for any other communications) - `cp_host:cp_port` is either an address or a name of the CP gRPC server and the port it is listening on - `lb_id` is the identifier of the virtual load balancer that was reserved (discussed below) - `data_host` and optional `data_port` are the address (hostnames are not allowed) of the Load Balancer - this is where the segments of events are sent to. When not specified, `data_port` defaults to 19522. Tyically the URI may contain two `data=` elements - one specifying IPv4 and the other - IPv6 address of the Load Balancer. - `sync_addr:sync_port` is the address of the Control Plane where UDP Sync packets must be sent by the sending site. Note that this is not required to be the same as the hostname or address used for gRPC communications. - `session_id` is the session ID issued when a worker is registered, see discussion below. The figures below show the two most common forms for the EJFAT URI: Amin URI - provided to you by the EJFAT team ![EJFAT URI](fig/AdminURI.png) Instance URI - returned to you when you reserve a Load Balancer ![EJFAT URI](fig/EJFATURI.png) Different control plane commands use different portions of the EJFAT URI and this section attempts to explain which portions of the URI is relevant at different stages as it relates to the diagram above. Step 1 (ReserveLoadBalancer): `ejfat[s]://@:/` Step 2 (Register worker node): `ejfat[s]://@:/lb/` - instance token and lbid come from Step 1 - Note that can be done from worker node or centrally by the WMS Steps 3a, 3b, 4, 7, and 8 (Sender sending data and sync packets and using AddSender/RemoveSender gRPC): `ejfat[s]://@:/lb/?sync=:&data=[&data=]` - Note that instance token and CP name and address and lb id aren't actually used, unless the uses Steps 7,8 Add/Remove sender gRPC calls - Note that up to two dataplane addresses of the loadbalancer (where to send the data) can be specified - an IPv4 and an IPv6 address - Note that 7 and 8 (addSenders/removeSenders) can be done from sender or centrally by the WMS Step 5 (SendState from worker node): `ejfat[s]://@:/lb/?sessionid=` - session id and session token come from the return of Step 2 - sync and data addresses are not relevant Step 6 (Deregister worker): `ejfat[s]://@:/lb/?sessionid=` - session id and session token come from Step 2 - sync and data addresses are not relevant - Note that can be done from worker node or centrally by WMS Step 9 (FreeLoadBalancer): `ejfat[s]://@:/lb//` Steps X and Y (GetLoadBalancer and LoadBalancerStatus): `ejfat[s]://@:/lb/` - instance token and lbid come from Step 1 Step Z (Overview): `ejfat[s]://@:/` - same as in Step 1 Note that `EjfatURI` object in the [code](https://github.com/JeffersonLab/E2SAR/blob/774a634078c8d6c0c096e28d07c60c8f3a12f53f/src/e2sarUtil.cpp#L47) can maintain multiple tokens internally and it knows which token to use in which sitation when using gRPC. Classes Segmenter and Reassembler also know how to ask `EjfatURI` object for the information they need. When loading a new EjfatURI object from a URI string (or environment variable) the constructor has a flag telling the object how to interpret the passed in token (admin, instance or session). ## Token administrative domains and security information flows EJFAT presumes multiple deployment options with different administrative entities being potentially responsible for managing a load-balancer farm, instances of virtual loadbalancers instantiated on that farm and even individual worker nodes communicating with a specific previously provisioned LB instance. The figure below attempts to capture these possibilities. The authority to perform certain operations is rooted in the possession of an admin, instance or session token. ![EJFAT Authority flows](fig/authority-flows.png) ## IP Address Use This section is informational use only and is not authoritative. See [UDPLBd documentation](https://github.com/esnet/udplbd) for more. Each load balancer is issued a pool of N (currently N <= 8) - IPv4 addresses - IPv6 addresses - UDP ports for Sync messages In addition the control plane uses one or two IP addresses - one for receiving Sync messages (on one of the ports) and one for communicating over gRPC, although under a typical installation those two are the same. This way each virtual load balancer (up to 8) reserved and allocated on the hardware gets an IPv4 address, an IPv6 address and a UDP port for Sync messages. The FPGA firmware configures the IPv4 and IPv6 addresses once the virtual instance is created, such that the instantiated load balancer can respond to ICMP requests on those, for example. Thus when reserving a new instance, the EjfatURI for it, as returned by gRPC `reserve()` call, reflects that information like shown in this typical example: ``` ejfats://@:/lb/?data=&data=&sync=: ``` ## Timings and other message rules EJFAT Control plane has a few timing rules and best practices for implementation: - Sync messages should be sent about 1 time per second by the sender (can be more often). The control plane uses a ringbuffer to calculate the rate slope. E2SAR also includes a low-pass filter for this purpose (you can set the sync period and the number of sync periods over which the reported rate is computed). - After a worker is registered, `sendState` must be sent within 10 seconds. In general a period of 100ms is recommended for doing `sendState` - When reserving a load balancer with `reserveLB` if the passed in duration time is 0 (i.e. UNIX epoch) the reservation never expires. ## Additional Information Additional details are contained in [this document](https://docs.google.com/document/d/1aUju_pWtHpS0Coesu8dC7HP6LbuKBJZqRYAMSSBtpWQ/edit#heading=h.zbhmzz3u1sna)