Fast Data Control Plane v2 Documentation

docs/products/fast_data_v2/overview.md

@@ -6,7 +6,7 @@ sidebar_label: Fast Data v2
 
 In today's rapidly evolving business landscape, organizations demand more than just data aggregation—they need **real-time responsiveness**, **enterprise-scale performance**, and **maximum flexibility** to adapt to business changing requirements.
 
-**Fast Data v2** addresses these needs by:
+**Fast Data v2** has been designed to outperform Fast Data v1 and to address these needs by:
 
 - **Accelerating Time-to-Value**: Deliver **up to 10x performance improvement**, significantly reducing real-time data aggregation processing and initial load times
 - **Enabling Real-Time Operations**: 24/7 data availability with minimal latency and less infrastructure resources utilization
@@ -39,12 +39,5 @@ To discover more about Fast Data v2 and understand how to leverage its capabilit
 
 1. **[Concepts](/products/fast_data_v2/concepts.mdx)** - Understand the foundational concepts, including the Fast Data message format and architectural principles
 2. **[Fast Data Engine](/products/fast_data_v2/fast_data_engine.md)** - Explore the four specialized workloads that power Fast Data v2 and learn how they work together to build high-performance data pipelines
-
-## Next Roadmap Steps
-
-Following the delivery of the Fast Data Engine v2, additional features and capabilities are in roadmap for future releases:
-
-- **Control Plane Runtime Management**: Enable full support of Fast Data runtime management capabilities provided by compatibility with [Fast Data v1 Control Plane](/products/fast_data/runtime_management/overview.mdx),
-  providing comprehensive monitoring and control over your Fast Data operations at runtime.
-- **Data Pipeline Configurator**: An upcoming UX/UI canvas-based configurator will allow you to
-  build data pipelines visually, further streamlining the development experience and boost interoperation and synergies with other Mia-Platform product suite components.
+3. **[Architecture](/products/fast_data_v2/architecture.md)** - Discover the most common examples of Fast Data architecture patterns, in order to compose modular data pipelines tailored to your specific business needs
+4. **[Runtime Management](/products/fast_data_v2/runtime_management/overview.mdx)** - Learn how to manage and control Fast Data execution at runtime, from the dedicated Control Plane UI

docs/products/fast_data_v2/runtime_management/application_configuration.md

@@ -0,0 +1,176 @@
+---
+id: application_configuration
+title: Application Configuration
+sidebar_label: Application Configuration
+---
+
+## Application Configuration
+
+In order to configure the Fast Data Control Plane application in your Fast Data Project, enter the Project Design Area, and find it inside the Application section.
+
+![Control Plane Fast Data Application Item](img/control-plane-app-item.png)
+
+The application is composed by 3 services (Control Plane, Control Plane Frontend, Envoy API Gateway), and 2 endpoints. Follow the creation wizard to instantiate these components in few clicks.
+
+:::note
+In case the application is not available in your Marketplace, please contact your Console administrator to verify if the Fast Data license is active for your Company.
+:::
+
+In the following paragraphs, you can find the guidelines to configure the Fast Data Control Plane resources to deploy them on your namespace.
+
+### Control Plane
+
+Once you have successfully created the application in the Design Area, move to the Control Plane microservice detail page.
+
+#### Configuration File
+
+Fill the `config.json` file of the `piper-configuration` configMap with the correct MongoDB connection.  
+To know more how to set properly the connection configuration properties, visit the [secret resolution](/products/fast_data_v2/secrets_resolution.md) documentation page.
+
+Control Plane microservice handles and persists in a MongoDB collection the desired states of each Fast Data pipeline step. This persistence layer allows to retrieve in every moment the last saved runtime states for your Fast Data pipeline even in worst cases of service restarts or failures.
+
+#### Required Kubernetes Permissions
+
+The Control Plane service requires specific RBAC permissions to automatically discover and manage Fast Data workloads in your namespace. Follow these steps to configure the necessary permissions:
+
+**Step 1: Identify the Service Account**
+
+Identify the service account name within the Control Plane microservice detail page. According the application setup, this name corresponds to the microservice itself, resulting in `control-plane`.
+
+**Step 2: Create the RBAC Configuration**
+
+Create two separate YAML files in your project Git repository to define the Role and RoleBinding:
+
+**`control-plane-role.yaml` file**
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: control-plane-role
+rules:
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["apps"]
+    resources: ["deployments"]
+    verbs: ["get", "list", "watch"]
+```
+
+**`control-plane-role-binding.yaml` file**
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: control-plane-role-binding
+subjects:
+  - kind: ServiceAccount
+    name: control-plane
+    namespace: {KUBE_NAMESPACE}
+roleRef:
+  kind: Role
+  name: control-plane-role
+  apiGroup: rbac.authorization.k8s.io
+```
+
+Commit these two files inside the `overlays/<YOUR_ENV>` project git folder, according to the specific runtime environment in which Fast Data control plane, as well as Fast Data workloads, will be deployed.
+
+:::note
+Note that these two YAML files must be committed **in each runtime environment** of your project in which we want to deploy the Fast Data Control Plane.
+:::
+
+**Step 3: Apply the Configuration**
+
+During the deployment process, the files will be automatically applied to the according namespace, and the `{KUBE_NAMESPACE}` placeholder will be automatically replaced with your project's actual Kubernetes namespace.
+
+Without these permissions, the Control Plane service cannot discover Fast Data workloads (Mongezium, Stream Processor, Farm Data, Kango) in your namespace and the Control Plane Frontend will not display your pipeline.
+
+#### In-Memory Storage
+
+You can opt for in-memory storage for your pipeline runtime states instead of persisting them.  
+To do so, change the control plane dockerImage by removing the `-mongodb` suffix from the image (e.g., use `/data-fabric/piper:0.1.1` instead of `/data-fabric/piper:0.1.1-mongodb`).
+
+:::caution
+Please note that in-memory storage is volatile. Unlike the persistence layer, it **does not support state recovery**; any service restart or failure will result in the **permanent loss** of all active pipeline runtime states.  
+The in-memory option is discouraged for production environments.
+:::
+
+### Control Plane Frontend
+
+The Control Plane Frontend enables to:
+
+- visualize pipelines structure and receive live updates about the runtime state of the Fast Data pipeline edges;
+- execute actions on pipelines state, that is pausing or resuming data consumption from one or more data streams
+
+The Control Plane Frontend is already pre-configured and ready to be deployed. No further user action is required to configure it.
+
+### Envoy API Gateway
+
+The Envoy API Gateway serves as the entry point for the Fast Data Control Plane application, providing essential routing and security capabilities.
+
+The Envoy API Gateway microservice is pre-configured and ready to be deployed. The routing configuration is automatically generated based on the endpoint definitions, and no manual intervention is required.
+
+#### Exposed Endpoints
+
+The application pre-configures two endpoints that are automatically routed by Envoy:
+
+- **`/api`** - Routes requests to the Control Plane service
+
+- **`/`** - Routes requests to the Control Plane Frontend service  
+
+## Workloads Configuration
+
+To enable proper communication between the Control Plane and the Fast Data workloads deployed in your namespace, each Fast Data workload must be configured to connect to the Control Plane.
+
+For each of the Fast Data workloads (Mongezium, Stream Processor, Farm Data, and Kango), you need to add the following configuration block to their respective JSON configuration files:
+
+```json
+{
+  "controlPlane": {
+    "grpcAddress": "http://control-plane:50051"
+  }
+}
+```
+
+:::note
+Without this configuration, the Fast Data workloads will operate independently and will not be visible or controllable through the Control Plane interface.
+:::
+
+### Additional Configuration Parameters
+
+Optionally, it is possible to add other parameters to the "control plane" config block, in particular:
+- **`resumeAfterMs`**: Time in milliseconds to wait before automatically resuming operations after a pause (15 seconds)
+- **`onCreate`**: Initial state when the workload for the first time is deployed and connects to the control plane. Available values are `"pause"` or `"resume"`. Set to `"pause"` to ensure the workload waits for explicit Control Plane commands from UI before start data consumption from data streams.
+
+:::note
+If no `onCreate` behavior is defined in the microservice ConfigMap, the **default runtime state** is **Running**. This design choice safeguards real-time data flows by preventing service interruptions when adding Control Plane connectivity to established Fast Data workloads.
+
+`onCreate` is applied solely if the Fast Data workload is the first time that is deployed. Otherwise, the parameter will be not considered by Control Plane, as it reads the last saved runtime state.
+:::
+
+```json
+{
+  "controlPlane": {
+    "grpcAddress": "http://control-plane:50051",
+    "resumeAfterMs": 15000,
+    "onCreate": "pause"
+  }
+}
+```
+
+## Application Deployment
+
+Once you have concluded all the above mentioned configuration steps, you can easily deploy your Project configuration!
+
+Verify the success of your deploy by:
+
+1. Opening the Control Plane Frontend UI
+2. Checking that all the configured Fast Data pipeline appears in the pipeline visualization
+3. Confirming that all the pipeline configuration details appears inside the details of each pipeline artifact
+4. Verifying that all execution steps implemented by Fast DAta workloads start in "paused" state as configured
+
+## Embed as Console Extension
+
+The Control Plane UI can be easily embedded as iFrame in either Overview Area or Runtime Area of your Fast Data Project.
+To know how to manage extensions in Console, visit the dedicated [documentation page](/products/console/company-configuration/extensions.md).

docs/products/fast_data_v2/runtime_management/best_practices.md

@@ -0,0 +1,65 @@
+---
+id: best_practices
+title: Best Practices
+sidebar_label: Best Practices
+---
+
+This page provides best practices and operational strategies for effectively managing Fast Data v2 pipelines using the Control Plane UI runtime controls.
+
+## Development Data Pipelines Best Practices
+
+### Visualize Fast Data Pipelines while Building Them
+
+During the Fast Data development phase, users can iteratively configure and continuously deploy in the development environment new Fast Data pipeline steps. Control Plane UI will provide the new architecture steps incrementally rendered, offering immediate visual feedback as the pipeline evolves.
+
+### Performance Testing and Simulation
+
+During the Fast Data development phase, users can simulate different scenarios for performance testing by pausing and resuming messages consumption along the pipeline. In this way, user can pause and resume operations to test system behavior under different load patterns before to promote to production.
+
+## Operational Management Strategies
+
+### Initial Load and Full Refresh Processes Management
+
+The **Control Plane UI** allows you to govern and orchestrate every stage of **Initial Load** or **Full Refresh** operations with precision and zero friction.
+
+#### 1. Controlled Initialization
+
+To ensure a stable start, every Fast Data workload can be configured with a default **Paused** runtime state. This is managed via the **`onCreate`** parameter within each microservice's **ConfigMap**. By initializing flows in a paused state, you ensure that no workload begins consuming data immediately after deployment, allowing for manual orchestration.
+
+#### 2. Ingestion and Lag Monitoring
+
+Once the environment is ready, you can initiate message loading into the ingestion layer of your pipeline. As the queues fill, the Control Plane provides real-time visibility into **Consumer Lag** across every pipeline edge, allowing you to monitor the volume of data awaiting processing.
+
+#### 3. Iterative Pipeline Activation
+
+After the initial data load, you can trigger consumption for the first stage of the pipeline using the **Play** button.
+
+* **Transformation Stage**: Typically, this first step involves executing transformation logic to ensure incoming data is compliant with Fast Data formats (e.g., casting, mapping, and data quality enhancements).
+* **Downstream Flow**: Once processed, these messages are produced into the output streams, ready for the subsequent stages of the pipeline.
+
+#### 4. Advanced Aggregation Management
+
+When dealing with **Aggregate execution steps**, the **Aggregation Graph Canvas** provides a centralized strategic view. This interface is specifically designed to manage complex scenarios where multiple data streams must be merged.
+
+**Best Practice: The Leaf-to-Head Strategy**
+For efficient ingestion, it is recommended to resume consumption following a "bottom-up" approach:
+
+1. **Start from the Leaves**: Resume consumption at the leaf nodes of the aggregation graph.
+2. **Monitor Lag**: Observe the incremental decrease in consumer lag.
+3. **Progression**: Once the lag approaches zero, move to the next level of the graph.
+4. **Activate the Head Node**: Finally, resume the head node of the aggregation.
+
+:::note
+By keeping the head node in a **Paused** state while the leaves process data, you prevent the production of premature events in the output stream. Once the head is resumed, it will produce the final aggregated output, significantly reducing redundant processing load on downstream stages.
+:::
+
+By combining real-time **Consumer Lag monitoring** with granular **runtime state control**, the Control Plane transforms complex Initial Load and Full Refresh operations into a manageable, transparent, and highly efficient process.
+
+### Strategic Resource Allocation and Performance Optimization
+
+By leveraging the ability to pause and resume message-consuming microservices in real-time, the Control Plane ensures that computing power is strategically directed toward high-priority tasks during peak demand periods. These granular runtime controls facilitate a balanced distribution of processing loads across every stage of the architecture, effectively mitigating bottlenecks and ensuring maximum resource utilization throughout your entire Fast Data v2 infrastructure.
+
+### Enhanced System Reliability
+
+When faced with scheduled maintenance or unforeseen anomalies, the Control Plane allows for precise intervention by pausing specific pipeline segments, ensuring that controlled troubleshooting occurs without compromising the broader system workflow.  
+This systematic approach extends into post-maintenance phases, where operations can be resumed gradually to verify stability and minimize recovery time. Beyond routine maintenance, these runtime controls facilitate effective fault isolation, enabling you to contain issues within localized segments to protect the integrity of the overall infrastructure. By implementing graceful degradation through precise shutdown and startup procedures, you ensure that your Fast Data v2 environment maintains absolute operational integrity even in challenging circumstances.

docs/products/fast_data_v2/runtime_management/compatibility_matrix.md

@@ -0,0 +1,47 @@
+---
+id: compatibility_matrix
+title: Control Plane Compatibility Matrix
+sidebar_label: Compatibility Matrix
+---
+
+This section guides you to understand whether your Project can support Fast Data v2 Control Plane and its features.
+
+## Infrastructure
+
+Here is described the compatibility between Fast Data v2 Control Plane application components and the external systems they rely on.
+Please ensure that versions shown in the matrix are respected in your deployed environments.
+
+| Service                 | Version | MongoDB |
+|-------------------------|---------|---------|
+| Control Plane           | 0.1.0   | \>=7.0  |
+
+## Fast Data Services
+
+Here is provided the compatibility matrix between Control Plane components and the Fast Data Engine v2 workloads that need to communicate with them.
+Please ensure that your services respect the following matrix.
+
+| [Control Plane Frontend](/products/fast_data_v2/runtime_management/application_configuration.md) | [Control Plane Piper](/products/fast_data_v2/runtime_management/application_configuration.md) | [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md) | [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md) | [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) | [Kango](/products/fast_data_v2/kango/10_Overview.md) |
+|:----------------------:|:-------------------:|:----------------------------------------------------------:|:---------------------------------------------------------------------:|:------------------------------------------------------:|:----------------------------------------------:|
+|         0.1.0          |        0.1.0        |                          \>=0.5.0                          |                                \>=0.6.0                                |                        \>=0.6.0                        |                    \>=0.6.0                    |
+
+## Service Latest Versions
+
+| Service                                                                                       | Version |
+|-----------------------------------------------------------------------------------------------|---------|
+| [Control Plane Frontend](/products/fast_data_v2/runtime_management/application_configuration.md)        | 0.1.0   |
+| [Control Plane Piper](/products/fast_data_v2/runtime_management/application_configuration.md)  | 0.1.0   |
+| [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md)                                   | 0.5.0   |
+| [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md)                         | 0.6.0   |
+| [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md)                                       | 0.6.0   |
+| [Kango](/products/fast_data_v2/kango/10_Overview.md)                                               | 0.6.0   |
+
+## Internal Compatibility
+
+| Service                    | Control Plane Frontend | Control Plane Piper | Mongezium | Stream Processor | Farm Data | Kango |
+|---------------------------:|:----------------------:|:-------------------:|:---------:|:----------------:|:---------:|:-----:|
+| Control Plane Frontend - 0.1.0 |         _N/A_          |        0.1.0        |   0.5.0   |      0.6.0       |   0.6.0   | 0.6.0 |
+| Control Plane Piper - 0.1.0    |         0.1.0          |        _N/A_        |   0.5.0   |      0.6.0       |   0.6.0   | 0.6.0 |
+| Mongezium - 0.5.0               |         0.1.0          |        0.1.0       |   _N/A_   |      0.6.0       |   0.6.0   | 0.6.0 |
+| Stream Processor - 0.6.0        |         0.1.0         |        0.1.0       |   0.5.0   |      _N/A_       |   0.6.0   | 0.6.0 |
+| Farm Data - 0.6.0               |         0.1.0          |        0.1.0        |   0.5.0   |      0.6.0       |   _N/A_   | 0.6.0 |
+| Kango - 0.6.0                   |         0.1.0          |        0.1.        |   0.5.0   |      0.6.0       |   0.6.0   | _N/A_ |