efa: Add support for 128-byte WQE#1708
Open
YonatanNachum wants to merge 11 commits intolinux-rdma:masterfrom
Open
efa: Add support for 128-byte WQE#1708YonatanNachum wants to merge 11 commits intolinux-rdma:masterfrom
YonatanNachum wants to merge 11 commits intolinux-rdma:masterfrom
Conversation
To commit: e736a223ab15 ("RDMA/efa: Expose new extended max inline buff size").
Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Store the newly added max inline buffer size in EFA context. For compatibility, if the driver doesn't support the new extended inline size, set it to the legacy one. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
As preparation for 128-byte WQE, add a validation for the requested inline size compared to the device's max supported inline size. We must do this validation prior to calculating the WQE size. Use the newly extended inline size which fallback to legacy value if driver didn't support query it. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
As preparation for supporting 128-byte WQE, remove direct usage of struct efa_io_tx_wqe wherever possible in WQE setters. This gives the ability to reuse those structs when needed for different WQE formats. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
As preparation for 128-byte WQE support, add a new SQ WQE size field to the SQ struct to store the dynamically configured WQE size. Change static usage of sizeof of the 64-byte WQE size with the new field. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Add the new 128-byte WQE structs and add the 128-byte WQE size to the SQ WQE calculation utility. For our datapath the logic is different between old and new APIs: 1. Old API - we use a buffer large enough for both WQEs and cast the buffer to the right pointers based on the WQE size. 2. New API - To have generic setters in the QP function pointers, add a new TX WQE context struct that holds the building blocks of the WQE and set them to the right fields for the WQE being constructed. This way the setters in the flow can access the fields in a generic way without being aware of the WQE type ore its size. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Add inline support for RDMA write with 128-byte WQE. To support write with inline ULP must provide a new flag in EFA create QP DV since EFA support write with inline only with 128-byte WQE. When the flag is provided 128-byte WQE is used for any requested inline size. Assign the inline buffer address in the SQ field to enable inline data setters for 128-byte WQE and set the remote length the same as local length. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
Report the new extended max inline size in EFA query device DV. The new inline size deprecates the legacy one and should be used from now. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
With 128-byte WQE support, the max SQ depth is half the size compared to when using 64-byte WQE. Add an EFA DV that given SQ init parameters, calculates the max SQ depth so ULPs can use it to create QPs without failing on SQ depth. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
EFA RQ max depth reported is the max descriptors a single RQ supports. For RQ EFA support multiple SGEs which means a single recv WR consumes multiple descriptors. This means that if the ULP requests more than 1 descriptor per recv WR, it can't use the max RQ depth reported. To help ULPs, add a new EFA DV that given the max recv SGEs requested, reports back the max RQ depth. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
EFA RQ max depth reported is the max descriptors a single RQ supports. For RQ EFA support multiple SGEs which means a single recv WR consumes multiple descriptors. This means that if the ULP requests more than 1 descriptor per recv WR, it can't use the max RQ depth reported. Validate the requested RQ depth relatively to the supported max RQ depth with the max SGEs requested. Signed-off-by: Yonatan Nachum <ynachum@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The EFA provider currently uses 64-byte WQEs supporting inline up to 32 bytes and only in send messages. This series adds support for 128-byte WQEs, which increases inline buffer size and enables inline support for RDMA write operations.
When 128-byte WQE is requested, the max SQ depth is cut in half. To expose the max SQ/RQ sizes supported given init attributes, we provide new DVs for users to query the supported max depth given those attributes.