Skip to content

Bindings

Jump to bottom
Xinxin Mei edited this page Mar 22, 2025 · 10 revisions

Python Bindings

The Python bindings for E2SAR are implemented with pybind11. You can find the source code in the 'src/pybind' directory.

Usage

The pybind11 module, named e2sar_py, is built alongside the C++ library and is named e2sar_py.cpython-<platform>.so based on the platform. To use the module, add its path to the Python environment via sys.path.append() with Python or export PYTHONPATH=<path> with Bash.

# Add the path of the pybind module
import sys
sys.path.append('/path/to/e2sar_py.cpython-<platform>.so')

# Import the module
import e2sar_py
# Your Python code here

Examples

We provide Jupyter notebooks in the 'scripts/notebooks/pybind11_examples' directory and the test suites to demonstrate the Python interfaces.

  • example_EjfatURI.ipynb: Demonstrates how to create Python EjfatURI classes and handle cases where C++ functions return result<T> types.
  • test_b2b_DP.py: demonstrate how to use Segmenter and Reassembler back-to-back on the local host.

Binding overview

We provide Python bindings for the following C++ classes. To ensure better organization, the compiled Python module e2sar_py includes two submodules: ControlPlane and DataPlane.

C++ class Python class
boost::asio::ip::address e2sar_py.IPAddress
e2sar::E2SARErrorc e2sar_py.E2SARErrorc
e2sar::E2SARErrorInfo e2sar_py.E2SARErrorInfo
e2sar::REHdr e2sar_py.REHdr
e2sar::LBHdr e2sar_py.LBHdr
e2sar::SyncHdr e2sar_py.SyncHdr
e2sar::e2sar_py.Affinity e2sar_py.Affinity
e2sar::EjfatURI e2sar_py.EjfatURI
e2sar::LBManager e2sar_py.ControlPlane.LBManager
e2sar::Segmenter e2sar_py.DataPlane.Segmenter
e2sar::Reassembler e2sar_py.DataPlane.Reassembler

Sending and receiving data with Segmenter and Reassembler

Segmenter

The underlying C++ Segmenter uses u_int8_t* pointers to reference data buffers, a type not directly supported in Python. We utilize the Python buffer protocol provided by pybind11's py::buffer to bridge the gap.

When sending data from Python, wrap your data in a buffer-compatible type. Here are two common approaches:

  • Using Python's built-in buffer-compatible types:

    Suitable built-in types include bytes, bytearray, and memoryview. You can use the following helper function to check if your object supports the buffer protocol:

    def supports_buffer_protocol(obj):
    try:
        memoryview(obj)
        return True
    except TypeError:
        return False

# Examples:
print(supports_buffer_protocol(b'abc'))                   # True
print(supports_buffer_protocol(bytearray(b'abc')))        # True
print(supports_buffer_protocol(memoryview(b'abc')))       # True
print(supports_buffer_protocol([1,2,3]))                  # False
print(supports_buffer_protocol("Hello"))                  # False

  • Using NumPy arrays:

    NumPy arrays naturally support the buffer protocol and can represent multi-dimensional data efficiently. When using NumPy arrays, ensure we explicitly inform the Segmenter of the total data length (in bytes) being sent.

Reassmbler

The C++ Reassembler::getEvent and Reassembler::recvEvent methods retrieve event data as raw pointers (uint8_t **). We wrap the raw pointer to Python bytes objects or numpy arrays.

Below are the Python binding signatures for Reassembler::getEvent. Both signatures refer to the same underlying C++ method. By default, the method returns data as Python bytes. However, if the context is known to be a numPy array and the data type is specified, you can use the numPy interface, which provides elements structured according to their data types.

An identical set of signatures also applies to Reassembler::recvEvent.

  • Get data as Python bytes:

    def getEventBytes() -> Tuple[int, bytes, int, int]:
    """
    Retrieves event data along with metadata.

    Returns:
        Tuple[int, bytes, int, int]:
            - Received length in bytes (int).
              - Returns -2 if an error occurs during retrieval.
              - Returns -1 if the buffer is empty (no message received).
            - Event data (bytes).
            - Event number (int).
            - Data identifier (int).
    """
    pass

  • Get data as a 1D NumPy array:

    def get1DNumpyArray(data_type: np.dtype) -> Tuple[int, np.ndarray, int, int]:
    """
    Retrieves an event from the Reassembler EventQueue as a 1D NumPy array.

    Args:
        data_type (np.dtype): Desired data type for the returned NumPy array.

    Returns:
        Tuple[int, np.ndarray, int, int]:
            - Event length in bytes (int), or negative error code.
            - Event data as a NumPy array (np.ndarray).
            - Event number (int).
            - Data identifier (int).
    """
    pass

Binding details

We bind the C++ functions to Python according to their return types. Our goal is to ensure that most of the C++ public methods and structs are covered, and that the return types in Python match their corresponding C++ return types. For more details, please refer to the pybind11 built-in type conversion guide.

However, there are a few specific exceptions:

  • C++ functions returning result<boost::tuple> are directly converted to Python tuples.
  • Methods like Reassembler::recvEvent and Reassembler::getEvent also return Python tuples directly.
  • The asynchronous method Segmenter::addToSendQueue with callback.

To quickly inspect available methods and their signatures in Python, use:

dir(py_class)            # List available methods and attributes
help(py_class.function)  # Show detailed function signature and documentation

Support for the C++ Result<T> type

We bind the C++ result<T> type to different Python e2sar_py.E2SARResultxxx objects. Check py_e2sar.cpp for details. When retrieving the returned context from a underlying C++ result<T> function, follow this convention:

res = some_function()  # an e2sar_py.E2SARResultXXXX instance
if res.has_error():
  print(res.error())   # check the error message

ret = res.value()   # the context

Testing with pytest

We use pytest to test the Python functionality. Currently, we support two test suites:

  • unit: Tests individual functions and methods at small scope.
  • b2b: Tests the local back-to-back Segmenter & Reassembler without involving any mock or real Load Balancer (LB). It includes send-and-receive tests using strings and numpy arrays.
  • cp: Tests the Control Pplane functions against a real LB.
  • lb-b2b: Tests with a real LB in-the-loop on localhost.

We recommend launching the b2b test suites separately as it requires a certain amount of memory. Running it alongside other tests may cause resource contention or failures.

Clone this wiki locally