Bennaaym/yaml support

.github/workflows/yaml_support.yml

@@ -0,0 +1,20 @@
+on: [push]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.8]
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install yaml_support dependencies
+        run: |
+          pip install extensions/yaml_support
+      - name: Run yaml_support tests
+        run: |
+          python -m extensions.yaml_support.labgraph_monitor.tests.test_lg_monitor_api
+          python -m extensions.yaml_support.labgraph_yaml_parser.tests.test_lg_yaml_api

extensions/yaml_support/README.md

@@ -0,0 +1,230 @@
+# YAML Support for LabGraph
+
+This extension provides an API to generate a serialized version of the labgraph topology. The serialized graph topology can be used in different applications E.g: server-client communication or to get a simplified overview of the topology in case of complicated graphs.
+
+## Quick Start
+
+### Method 1 - building from source code
+
+**Prerequisites**:
+
+- Python3\
+  Supported python version(s)
+  _ [Python3.6](https://www.python.org/downloads/)
+  _ [Python3.8](https://www.python.org/downloads/) (**RECOMMENDED**)
+- Make sure to install [labgraph](https://github.com/facebookresearch/labgraph) before proceeding
+
+```
+cd labgraph/extensions/yaml_support
+python setup.py install
+```
+
+### Testing:
+
+To make sure things are working:
+
+1- Move to the root of the LabGraph directory:
+
+```
+labgraph\extensions\yaml_support> cd ../..
+labgraph>
+```
+
+2- Run the following tests
+
+```
+python -m extensions.yaml_support.labgraph_monitor.tests.test_lg_monitor_api
+```
+
+```
+python -m extensions.yaml_support.labgraph_yaml_parser.tests.test_lg_yaml_api
+```
+
+### API
+
+#### Labgraph Monitor:
+
+**generate_labgraph_monitor(graph: lg.Graph) -> None** : This function can be used to generate a serialized version of the passed graph instance. The serialized version of the graph will be streamed
+to the clients using LabGraph Websockets API.
+
+1. Call **generate_labgraph_monitor(graph: lg.Graph) -> None** and pass an instance of the graph as a parameter
+
+```
+from extensions.yaml_support.labgraph_monitor.generate_lg_monitor.generate_lg_monitor import (
+    generate_labgraph_monitor
+)
+
+generate_labgraph_monitor(graph)
+```
+
+This will start a websocket server on localhost port 9000 (127.0.0.1:9000)
+
+2. To start receiving data from the server, send the following `StartStreamRequest` to **ws://127.0.0.1:9000**
+
+```
+{
+   "api_version": "0.1",
+   "api_request": {
+     "request_id": 1,
+     "start_stream_request": {
+       "stream_id": "LABGRAPH.MONITOR",
+        "labgraph.monitor": {
+        }
+     }
+   }
+}
+```
+
+A serialized representation of the graph should be received each 200ms
+
+The graph representation has the following schema:
+
+```
+{
+    name: "graph_name",
+    nodes: {
+        "node_name":{
+            upstreams:{
+                "upstream_name":[
+                    {
+                        name: "message_name",
+                        type: "message_type",
+                    }
+                ]
+            }
+        }
+    }
+}
+```
+
+E.g:
+
+```
+{
+    "stream_batch": {
+        "stream_id": "LABGRAPH.MONITOR",
+        "labgraph.monitor": {
+            "samples": [
+                {
+                    "data": {
+                        "name": "Demo",
+                        "nodes": {
+                            "NoiseGenerator": {
+                                "upstreams": {}
+                            },
+                            "RollingAverager": {
+                                "upstreams": {
+                                    "NoiseGenerator": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ]
+                                }
+                            },
+                            "Amplifier": {
+                                "upstreams": {
+                                    "NoiseGenerator": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ]
+                                }
+                            },
+                            "Attenuator": {
+                                "upstreams": {
+                                    "NoiseGenerator": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ]
+                                }
+                            },
+                            "Sink": {
+                                "upstreams": {
+                                    "RollingAverager": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ],
+                                    "Amplifier": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ],
+                                    "Attenuator": [
+                                        {
+                                            "name": "RandomMessage",
+                                            "fields": {
+                                                "timestamp": "float",
+                                                "data": "ndarray"
+                                            }
+                                        }
+                                    ]
+                                }
+                            }
+                        }
+                    },
+                    "produced_timestamp_s": 1644931422.141309,
+                    "timestamp_s": 1644931422.141309
+                }
+            ],
+            "batch_num": 54
+        }
+    }
+}
+```
+
+The above-serialized representation was generated for the following graph:
+https://github.com/facebookresearch/labgraph/blob/main/extensions/graphviz_support/graphviz_support/tests/demo_graph/demo.py
+
+3. To stop receiving data from the server, send the following `EndStreamRequest` to **ws://127.0.0.1:9000**
+
+```
+{
+   "api_version": "0.1",
+   "api_request": {
+     "request_id": 1,
+     "end_stream_request": {
+       "stream_id": "LABGRAPH.MONITOR",
+        "labgraph.monitor": {
+        }
+     }
+   }
+}
+```
+
+#### Labgraph YAML Parser:
+
+**yamlify(python_file: str, yaml_file: str) -> str** : This function can be used to generate a YAMLIFIED version of the passed LabGraph source code(.py). The serialized version will be saved as a YAML file at the specified folder(yml_file)
+
+1. Call **yamlify(python_file: str, yaml_file: str) -> str** and pass the appropriate parameters
+
+```
+from extensions.yaml_support.labgraph_yaml_parser.yamlify import (
+    yamlify
+)
+
+yamlify(python_file, output_yaml_file)
+```
+
+This will generate a YAML file in the specified location

extensions/yaml_support/labgraph_monitor/aliases/aliases.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+# Copyright 2004-present Facebook. All Rights Reserved.
+
+from typing import Dict, List, Union
+
+"""
+{
+    name: "message_name",
+    fields: {
+            "field_name": "field_type",
+            ...
+    }
+}
+"""
+
+SerializedMessage = Dict[
+    str,
+    Union[
+        str,
+        Dict[str, str]
+    ]
+]
+
+
+"""
+{
+    name: "graph_name",
+    nodes: {
+        "node_name":{
+            upstreams:{
+                "upstream_name":[
+                    SerializedMessage,
+                ],
+                ...
+            }
+        }
+    }
+}
+"""
+SerializedGraph = Dict[
+    str,
+    Union[
+        str,
+        Dict[
+            str,
+            Dict[
+                str,
+                Dict[
+                    str,
+                    List[SerializedMessage]
+                ]
+            ]
+        ]
+    ]
+]