diff --git a/docs/products/fast_data_v2/overview.md b/docs/products/fast_data_v2/overview.md index ca26cc5ee7..df342d9db9 100644 --- a/docs/products/fast_data_v2/overview.md +++ b/docs/products/fast_data_v2/overview.md @@ -6,7 +6,7 @@ sidebar_label: Fast Data v2 In today's rapidly evolving business landscape, organizations demand more than just data aggregation—they need **real-time responsiveness**, **enterprise-scale performance**, and **maximum flexibility** to adapt to business changing requirements. -**Fast Data v2** addresses these needs by: +**Fast Data v2** has been designed to outperform Fast Data v1 and to address these needs by: - **Accelerating Time-to-Value**: Deliver **up to 10x performance improvement**, significantly reducing real-time data aggregation processing and initial load times - **Enabling Real-Time Operations**: 24/7 data availability with minimal latency and less infrastructure resources utilization @@ -39,12 +39,5 @@ To discover more about Fast Data v2 and understand how to leverage its capabilit 1. **[Concepts](/products/fast_data_v2/concepts.mdx)** - Understand the foundational concepts, including the Fast Data message format and architectural principles 2. **[Fast Data Engine](/products/fast_data_v2/fast_data_engine.md)** - Explore the four specialized workloads that power Fast Data v2 and learn how they work together to build high-performance data pipelines - -## Next Roadmap Steps - -Following the delivery of the Fast Data Engine v2, additional features and capabilities are in roadmap for future releases: - -- **Control Plane Runtime Management**: Enable full support of Fast Data runtime management capabilities provided by compatibility with [Fast Data v1 Control Plane](/products/fast_data/runtime_management/overview.mdx), - providing comprehensive monitoring and control over your Fast Data operations at runtime. -- **Data Pipeline Configurator**: An upcoming UX/UI canvas-based configurator will allow you to - build data pipelines visually, further streamlining the development experience and boost interoperation and synergies with other Mia-Platform product suite components. +3. **[Architecture](/products/fast_data_v2/architecture.md)** - Discover the most common examples of Fast Data architecture patterns, in order to compose modular data pipelines tailored to your specific business needs +4. **[Runtime Management](/products/fast_data_v2/runtime_management/overview.mdx)** - Learn how to manage and control Fast Data execution at runtime, from the dedicated Control Plane UI diff --git a/docs/products/fast_data_v2/runtime_management/application_configuration.md b/docs/products/fast_data_v2/runtime_management/application_configuration.md new file mode 100644 index 0000000000..5aeb6eb455 --- /dev/null +++ b/docs/products/fast_data_v2/runtime_management/application_configuration.md @@ -0,0 +1,176 @@ +--- +id: application_configuration +title: Application Configuration +sidebar_label: Application Configuration +--- + +## Application Configuration + +In order to configure the Fast Data Control Plane application in your Fast Data Project, enter the Project Design Area, and find it inside the Application section. + +![Control Plane Fast Data Application Item](img/control-plane-app-item.png) + +The application is composed by 3 services (Control Plane, Control Plane Frontend, Envoy API Gateway), and 2 endpoints. Follow the creation wizard to instantiate these components in few clicks. + +:::note +In case the application is not available in your Marketplace, please contact your Console administrator to verify if the Fast Data license is active for your Company. +::: + +In the following paragraphs, you can find the guidelines to configure the Fast Data Control Plane resources to deploy them on your namespace. + +### Control Plane + +Once you have successfully created the application in the Design Area, move to the Control Plane microservice detail page. + +#### Configuration File + +Fill the `config.json` file of the `piper-configuration` configMap with the correct MongoDB connection. +To know more how to set properly the connection configuration properties, visit the [secret resolution](/products/fast_data_v2/secrets_resolution.md) documentation page. + +Control Plane microservice handles and persists in a MongoDB collection the desired states of each Fast Data pipeline step. This persistence layer allows to retrieve in every moment the last saved runtime states for your Fast Data pipeline even in worst cases of service restarts or failures. + +#### Required Kubernetes Permissions + +The Control Plane service requires specific RBAC permissions to automatically discover and manage Fast Data workloads in your namespace. Follow these steps to configure the necessary permissions: + +**Step 1: Identify the Service Account** + +Identify the service account name within the Control Plane microservice detail page. According the application setup, this name corresponds to the microservice itself, resulting in `control-plane`. + +**Step 2: Create the RBAC Configuration** + +Create two separate YAML files in your project Git repository to define the Role and RoleBinding: + +**`control-plane-role.yaml` file** + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: control-plane-role +rules: + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "list", "watch"] + - apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get", "list", "watch"] +``` + +**`control-plane-role-binding.yaml` file** + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: control-plane-role-binding +subjects: + - kind: ServiceAccount + name: control-plane + namespace: {KUBE_NAMESPACE} +roleRef: + kind: Role + name: control-plane-role + apiGroup: rbac.authorization.k8s.io +``` + +Commit these two files inside the `overlays/` project git folder, according to the specific runtime environment in which Fast Data control plane, as well as Fast Data workloads, will be deployed. + +:::note +Note that these two YAML files must be committed **in each runtime environment** of your project in which we want to deploy the Fast Data Control Plane. +::: + +**Step 3: Apply the Configuration** + +During the deployment process, the files will be automatically applied to the according namespace, and the `{KUBE_NAMESPACE}` placeholder will be automatically replaced with your project's actual Kubernetes namespace. + +Without these permissions, the Control Plane service cannot discover Fast Data workloads (Mongezium, Stream Processor, Farm Data, Kango) in your namespace and the Control Plane Frontend will not display your pipeline. + +#### In-Memory Storage + +You can opt for in-memory storage for your pipeline runtime states instead of persisting them. +To do so, change the control plane dockerImage by removing the `-mongodb` suffix from the image (e.g., use `/data-fabric/piper:0.1.1` instead of `/data-fabric/piper:0.1.1-mongodb`). + +:::caution +Please note that in-memory storage is volatile. Unlike the persistence layer, it **does not support state recovery**; any service restart or failure will result in the **permanent loss** of all active pipeline runtime states. +The in-memory option is discouraged for production environments. +::: + +### Control Plane Frontend + +The Control Plane Frontend enables to: + +- visualize pipelines structure and receive live updates about the runtime state of the Fast Data pipeline edges; +- execute actions on pipelines state, that is pausing or resuming data consumption from one or more data streams + +The Control Plane Frontend is already pre-configured and ready to be deployed. No further user action is required to configure it. + +### Envoy API Gateway + +The Envoy API Gateway serves as the entry point for the Fast Data Control Plane application, providing essential routing and security capabilities. + +The Envoy API Gateway microservice is pre-configured and ready to be deployed. The routing configuration is automatically generated based on the endpoint definitions, and no manual intervention is required. + +#### Exposed Endpoints + +The application pre-configures two endpoints that are automatically routed by Envoy: + +- **`/api`** - Routes requests to the Control Plane service + +- **`/`** - Routes requests to the Control Plane Frontend service + +## Workloads Configuration + +To enable proper communication between the Control Plane and the Fast Data workloads deployed in your namespace, each Fast Data workload must be configured to connect to the Control Plane. + +For each of the Fast Data workloads (Mongezium, Stream Processor, Farm Data, and Kango), you need to add the following configuration block to their respective JSON configuration files: + +```json +{ + "controlPlane": { + "grpcAddress": "http://control-plane:50051" + } +} +``` + +:::note +Without this configuration, the Fast Data workloads will operate independently and will not be visible or controllable through the Control Plane interface. +::: + +### Additional Configuration Parameters + +Optionally, it is possible to add other parameters to the "control plane" config block, in particular: +- **`resumeAfterMs`**: Time in milliseconds to wait before automatically resuming operations after a pause (15 seconds) +- **`onCreate`**: Initial state when the workload for the first time is deployed and connects to the control plane. Available values are `"pause"` or `"resume"`. Set to `"pause"` to ensure the workload waits for explicit Control Plane commands from UI before start data consumption from data streams. + +:::note +If no `onCreate` behavior is defined in the microservice ConfigMap, the **default runtime state** is **Running**. This design choice safeguards real-time data flows by preventing service interruptions when adding Control Plane connectivity to established Fast Data workloads. + +`onCreate` is applied solely if the Fast Data workload is the first time that is deployed. Otherwise, the parameter will be not considered by Control Plane, as it reads the last saved runtime state. +::: + +```json +{ + "controlPlane": { + "grpcAddress": "http://control-plane:50051", + "resumeAfterMs": 15000, + "onCreate": "pause" + } +} +``` + +## Application Deployment + +Once you have concluded all the above mentioned configuration steps, you can easily deploy your Project configuration! + +Verify the success of your deploy by: + +1. Opening the Control Plane Frontend UI +2. Checking that all the configured Fast Data pipeline appears in the pipeline visualization +3. Confirming that all the pipeline configuration details appears inside the details of each pipeline artifact +4. Verifying that all execution steps implemented by Fast DAta workloads start in "paused" state as configured + +## Embed as Console Extension + +The Control Plane UI can be easily embedded as iFrame in either Overview Area or Runtime Area of your Fast Data Project. +To know how to manage extensions in Console, visit the dedicated [documentation page](/products/console/company-configuration/extensions.md). diff --git a/docs/products/fast_data_v2/runtime_management/best_practices.md b/docs/products/fast_data_v2/runtime_management/best_practices.md new file mode 100644 index 0000000000..edcf912f9a --- /dev/null +++ b/docs/products/fast_data_v2/runtime_management/best_practices.md @@ -0,0 +1,65 @@ +--- +id: best_practices +title: Best Practices +sidebar_label: Best Practices +--- + +This page provides best practices and operational strategies for effectively managing Fast Data v2 pipelines using the Control Plane UI runtime controls. + +## Development Data Pipelines Best Practices + +### Visualize Fast Data Pipelines while Building Them + +During the Fast Data development phase, users can iteratively configure and continuously deploy in the development environment new Fast Data pipeline steps. Control Plane UI will provide the new architecture steps incrementally rendered, offering immediate visual feedback as the pipeline evolves. + +### Performance Testing and Simulation + +During the Fast Data development phase, users can simulate different scenarios for performance testing by pausing and resuming messages consumption along the pipeline. In this way, user can pause and resume operations to test system behavior under different load patterns before to promote to production. + +## Operational Management Strategies + +### Initial Load and Full Refresh Processes Management + +The **Control Plane UI** allows you to govern and orchestrate every stage of **Initial Load** or **Full Refresh** operations with precision and zero friction. + +#### 1. Controlled Initialization + +To ensure a stable start, every Fast Data workload can be configured with a default **Paused** runtime state. This is managed via the **`onCreate`** parameter within each microservice's **ConfigMap**. By initializing flows in a paused state, you ensure that no workload begins consuming data immediately after deployment, allowing for manual orchestration. + +#### 2. Ingestion and Lag Monitoring + +Once the environment is ready, you can initiate message loading into the ingestion layer of your pipeline. As the queues fill, the Control Plane provides real-time visibility into **Consumer Lag** across every pipeline edge, allowing you to monitor the volume of data awaiting processing. + +#### 3. Iterative Pipeline Activation + +After the initial data load, you can trigger consumption for the first stage of the pipeline using the **Play** button. + +* **Transformation Stage**: Typically, this first step involves executing transformation logic to ensure incoming data is compliant with Fast Data formats (e.g., casting, mapping, and data quality enhancements). +* **Downstream Flow**: Once processed, these messages are produced into the output streams, ready for the subsequent stages of the pipeline. + +#### 4. Advanced Aggregation Management + +When dealing with **Aggregate execution steps**, the **Aggregation Graph Canvas** provides a centralized strategic view. This interface is specifically designed to manage complex scenarios where multiple data streams must be merged. + +**Best Practice: The Leaf-to-Head Strategy** +For efficient ingestion, it is recommended to resume consumption following a "bottom-up" approach: + +1. **Start from the Leaves**: Resume consumption at the leaf nodes of the aggregation graph. +2. **Monitor Lag**: Observe the incremental decrease in consumer lag. +3. **Progression**: Once the lag approaches zero, move to the next level of the graph. +4. **Activate the Head Node**: Finally, resume the head node of the aggregation. + +:::note +By keeping the head node in a **Paused** state while the leaves process data, you prevent the production of premature events in the output stream. Once the head is resumed, it will produce the final aggregated output, significantly reducing redundant processing load on downstream stages. +::: + +By combining real-time **Consumer Lag monitoring** with granular **runtime state control**, the Control Plane transforms complex Initial Load and Full Refresh operations into a manageable, transparent, and highly efficient process. + +### Strategic Resource Allocation and Performance Optimization + +By leveraging the ability to pause and resume message-consuming microservices in real-time, the Control Plane ensures that computing power is strategically directed toward high-priority tasks during peak demand periods. These granular runtime controls facilitate a balanced distribution of processing loads across every stage of the architecture, effectively mitigating bottlenecks and ensuring maximum resource utilization throughout your entire Fast Data v2 infrastructure. + +### Enhanced System Reliability + +When faced with scheduled maintenance or unforeseen anomalies, the Control Plane allows for precise intervention by pausing specific pipeline segments, ensuring that controlled troubleshooting occurs without compromising the broader system workflow. +This systematic approach extends into post-maintenance phases, where operations can be resumed gradually to verify stability and minimize recovery time. Beyond routine maintenance, these runtime controls facilitate effective fault isolation, enabling you to contain issues within localized segments to protect the integrity of the overall infrastructure. By implementing graceful degradation through precise shutdown and startup procedures, you ensure that your Fast Data v2 environment maintains absolute operational integrity even in challenging circumstances. diff --git a/docs/products/fast_data_v2/runtime_management/compatibility_matrix.md b/docs/products/fast_data_v2/runtime_management/compatibility_matrix.md new file mode 100644 index 0000000000..9e9caca165 --- /dev/null +++ b/docs/products/fast_data_v2/runtime_management/compatibility_matrix.md @@ -0,0 +1,47 @@ +--- +id: compatibility_matrix +title: Control Plane Compatibility Matrix +sidebar_label: Compatibility Matrix +--- + +This section guides you to understand whether your Project can support Fast Data v2 Control Plane and its features. + +## Infrastructure + +Here is described the compatibility between Fast Data v2 Control Plane application components and the external systems they rely on. +Please ensure that versions shown in the matrix are respected in your deployed environments. + +| Service | Version | MongoDB | +|-------------------------|---------|---------| +| Control Plane | 0.1.0 | \>=7.0 | + +## Fast Data Services + +Here is provided the compatibility matrix between Control Plane components and the Fast Data Engine v2 workloads that need to communicate with them. +Please ensure that your services respect the following matrix. + +| [Control Plane Frontend](/products/fast_data_v2/runtime_management/application_configuration.md) | [Control Plane Piper](/products/fast_data_v2/runtime_management/application_configuration.md) | [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md) | [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md) | [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) | [Kango](/products/fast_data_v2/kango/10_Overview.md) | +|:----------------------:|:-------------------:|:----------------------------------------------------------:|:---------------------------------------------------------------------:|:------------------------------------------------------:|:----------------------------------------------:| +| 0.1.0 | 0.1.0 | \>=0.5.0 | \>=0.6.0 | \>=0.6.0 | \>=0.6.0 | + +## Service Latest Versions + +| Service | Version | +|-----------------------------------------------------------------------------------------------|---------| +| [Control Plane Frontend](/products/fast_data_v2/runtime_management/application_configuration.md) | 0.1.0 | +| [Control Plane Piper](/products/fast_data_v2/runtime_management/application_configuration.md) | 0.1.0 | +| [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md) | 0.5.0 | +| [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md) | 0.6.0 | +| [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) | 0.6.0 | +| [Kango](/products/fast_data_v2/kango/10_Overview.md) | 0.6.0 | + +## Internal Compatibility + +| Service | Control Plane Frontend | Control Plane Piper | Mongezium | Stream Processor | Farm Data | Kango | +|---------------------------:|:----------------------:|:-------------------:|:---------:|:----------------:|:---------:|:-----:| +| Control Plane Frontend - 0.1.0 | _N/A_ | 0.1.0 | 0.5.0 | 0.6.0 | 0.6.0 | 0.6.0 | +| Control Plane Piper - 0.1.0 | 0.1.0 | _N/A_ | 0.5.0 | 0.6.0 | 0.6.0 | 0.6.0 | +| Mongezium - 0.5.0 | 0.1.0 | 0.1.0 | _N/A_ | 0.6.0 | 0.6.0 | 0.6.0 | +| Stream Processor - 0.6.0 | 0.1.0 | 0.1.0 | 0.5.0 | _N/A_ | 0.6.0 | 0.6.0 | +| Farm Data - 0.6.0 | 0.1.0 | 0.1.0 | 0.5.0 | 0.6.0 | _N/A_ | 0.6.0 | +| Kango - 0.6.0 | 0.1.0 | 0.1. | 0.5.0 | 0.6.0 | 0.6.0 | _N/A_ | diff --git a/docs/products/fast_data_v2/runtime_management/control_plane_ui.md b/docs/products/fast_data_v2/runtime_management/control_plane_ui.md new file mode 100644 index 0000000000..4f5c824eb3 --- /dev/null +++ b/docs/products/fast_data_v2/runtime_management/control_plane_ui.md @@ -0,0 +1,143 @@ +--- +id: control_plane_ui +title: Control Plane UI +sidebar_label: Control Plane UI +--- + +The Fast Data Control Plane UI provides a comprehensive visual interface for managing and monitoring your Fast Data v2 pipelines. +This web-based interface allows you to visualize the entire data pipeline architecture, monitor real-time performance, and control runtime states of your Fast Data Engine workloads with just a few clicks. + +## Artifacts + +Fast Data Control Plane shows various artifacts that represent different aspects of the overall architecture of Fast Data pipelines. + +### Data Streams + +Data Streams represent the channels through which data flows between different execution steps in your pipeline. In the Fast Data v2 architecture, these are Kafka topics that typically carry [Fast Data compliant messages](/products/fast_data_v2/concepts.mdx#fast-data-message-format) between workloads. + +Each Data Stream lists its associated consumers, detailing essential metadata such as consumer names, runtime states, and the consumer groups that define each Fast Data workload. + +### Persisted Assets + +Persisted Assets represent data that is stored persistently at a certain step of the pipeline. These can be often the final data products (Single Views) that result from your Fast Data pipeline operations. + +Each Persisted Asset displays useful metadat and other information about the persistor. + +## Execution Steps + +Execution Steps are the core processing components of your Fast Data pipeline, implemented by the four specialized workloads of the Fast Data Engine v2. Each step appears as a circular node in the pipeline diagram and represents a specific data processing operation. + +### Ingest + +The **Ingest** step is implemented by [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md), that acts as Change Data Capture (CDC) from MongoDB collections to Kafka streams. + +The Ingest step monitors MongoDB change streams in real-time and converts database operations (insert, update, delete) into Fast Data compliant messages that flow to downstream processing steps. + +When you click on an Ingest step, the detail panel shows two tabs: + +- **Output Topics**: Lists all Kafka topics where change events are published. Each topic corresponds to a MongoDB collection being monitored for changes. +- **Details**: Provides useful metadata like the type of the execution step, the version of the microservice that implements the steps and the number of replicas. + +### Process + +The **Process** step is implemented by [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md) and provides powerful data transformation capabilities using custom JavaScript logic executed in a secure sandbox environment. + +The Process step detail panel contains three tabs: + +- **Consumers**: Shows the input stream that the Stream Processor is consuming from, including consumer group information and current lag metrics. +- **Function**: Displays the custom JavaScript transformation logic being executed on the data stream. This tab shows the actual processing code that filters, transforms, maps, or validates incoming messages. +- **Details**: Provides essential metadata, including the execution step type, the microservice version, and the number of replicas, alongside configuration details such as caching settings and other key parameters. + +### Aggregate + +The **Aggregate** step is implemented by [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) and performs real-time multi-stream data aggregation to create data products. + +The Aggregate step offers three tabs in its detail panel: + +- **Consumers**: Lists all input data streams being consumed for aggregation, along with consumer lag and runtime state for each stream. +- **Aggregation**: Provides access to the aggregation graph canvas (see detailed explanation in the next paragraph). +- **Details**: Provides essential metadata, including the execution step type, the microservice version, and the number of replicas, alongside configuration details such as caching settings, internal updates topic settings and other key parameters. + +#### Aggregation Graph Canvas + +The Aggregation Graph Canvas is a specialized visual interface within the Aggregate step that displays the entity relationship diagram configured for your aggregation logic. This canvas shows how different data streams are structurally combined to produce the final aggregated output. + +The Aggregation Graph Canvas offers a comprehensive visual representation of entity relationships within the aggregation process, clearly illustrating how different data entities are related with each other to build the final aggregated output. This interface provides detailed information about relationship configurations that define how entities are joined and aggregated together. +The canvas enables interactive exploration of the entire aggregation logic, allowing you to select any two data streams in the graph to examine the specific relationship configuration that governs their interaction, including join conditions and aggregation rules. + +Moreover, from the Aggregation Graph Canvas, it is possible to control the runtime state for data consumption from the rendered data streams. For more information on how to control flows, proceed to the following paragraphs. + +### Persist + +The **Persist** step is implemented by [Kango](/products/fast_data_v2/kango/10_Overview.md) and handles the final storage of processed data from Kafka streams to MongoDB collections (Persisted Assets). + +The Persist step detail panel includes two tabs: + +- **Consumers**: Shows the input stream that Kango is consuming for persistence, including consumer group details and consumer lag metrics. +- **Details**: Provides useful metadata like the type of the execution step, the version of the microservice that implements the steps, the number of replicas and other information including connection name and the target MongoDB collection. + +## Runtime States + +Each execution step in your pipeline can be in one of four distinct runtime states, visually indicated by different colors and indicators in the Control Plane UI. + +### Running + +The **Running** state indicates that the workload's consumer is actively consuming data from the input stream. + +Running steps are typically indicated by blue dotted line in the UI. + +### Paused + +The **Paused** state means that the workload's consumer has stopped consuming data from its input streams. This is typically a user-initiated action for maintenance, testing, or controlled processing scenarios. + +In Paused state, the workload remains healthy and responsive, no new messages are being consumed from input streams; thus, in case of new input messages, the displayed consumer lag will consequently increase. + +Paused steps are typically indicated by grey line in the UI. + +### Unsubscribed + +The **Unsubscribed** state indicates that the workload's consumer is not subscribed to its input streams and therefore cannot consume any data. + +This commonly occurs when the consumer group has been unsubscribed from that topic, or when the workload has no replicas running (scaled down completely). + +In this state, the consumer cannot process any data and it is no more possible to interact with Play/Pause button and to know about consumer lag info. + +Unsubscribed steps are typically indicated by orange line in the UI. + +### Unknown + +The **Unknown** state indicates that the Control Plane cannot determine the current state of the workload. This typically happens when the service is not reachable or responsive, the workload has crashed or encountered fatal errors, network issues prevent state communication, the service is in an invalid or corrupted state. + +Unknown states require investigation and usually indicate operational issues that need immediate attention. + +Unknown steps are typically indicated by red line in the UI. + +## Control Runtime States + +The Control Plane UI provides powerful controls for managing the runtime state of your Fast Data pipeline steps, enabling sophisticated operational strategies for data processing. + +Each data flows inside the pipeline provides **Pause** and **Play** buttons that allow you to stop and start data consumption from the data stream. + +- **Pause**: Stops the consumer from processing new messages while maintaining stream position +- **Resume**: Restarts data consumption from the previously maintained position + +Pause and Play buttons are available whenever you click on a pipeline step that supports runtime state control for specific data flows. +Additionally, for the Aggregate execution step, these same controls are also available directly within the Aggregation Graph Canvas, providing enhanced utility for managing Initial Load and Full Refresh scenarios, allowing for more efficient and optimized runtime control in these and other operational scenarios. + +For more detailed operational strategies and best practices on using these runtime controls effectively, visit the [Best Practices documentation](/products/fast_data_v2/runtime_management/best_practices.md). + +## Navigating UI + +The Control Plane UI provides several features to help you efficiently navigate and interact with your pipeline visualization. + +### Search elements on the canvas + +The search feature enables you to quickly locate specific elements within your pipeline. + +Simply type on the searchbar the name of an artifact or Fast Data workload and select it. The interface will instantly focus on and highlight the selected element, automatically opening the contextual drawer to display all relevant information about it. + +### Highlight on Hover + +Typically, when you hover over elements listed in the detail panels or drawers, the corresponding involved pipeline elements are instantly highlighted inside the canvas. + +This highlighting feature significantly improves navigation efficiency, especially in complex pipelines with many interconnected components. diff --git a/docs/products/fast_data_v2/runtime_management/img/control-plane-app-item.png b/docs/products/fast_data_v2/runtime_management/img/control-plane-app-item.png new file mode 100644 index 0000000000..881a73e77a Binary files /dev/null and b/docs/products/fast_data_v2/runtime_management/img/control-plane-app-item.png differ diff --git a/docs/products/fast_data_v2/runtime_management/img/control-plane-v2-architecture.png b/docs/products/fast_data_v2/runtime_management/img/control-plane-v2-architecture.png new file mode 100644 index 0000000000..864fed48b1 Binary files /dev/null and b/docs/products/fast_data_v2/runtime_management/img/control-plane-v2-architecture.png differ diff --git a/docs/products/fast_data_v2/runtime_management/img/fast-data-pipeline.png b/docs/products/fast_data_v2/runtime_management/img/fast-data-pipeline.png new file mode 100644 index 0000000000..a5536246d1 Binary files /dev/null and b/docs/products/fast_data_v2/runtime_management/img/fast-data-pipeline.png differ diff --git a/docs/products/fast_data_v2/runtime_management/overview.mdx b/docs/products/fast_data_v2/runtime_management/overview.mdx new file mode 100644 index 0000000000..036c2de9d2 --- /dev/null +++ b/docs/products/fast_data_v2/runtime_management/overview.mdx @@ -0,0 +1,65 @@ +--- +id: overview +title: Runtime Management +sidebar_label: Runtime Management +--- + +# Runtime Management + +## Fast Data Control Plane + +Mia-Platform Fast Data includes a Runtime Management solution, the **Fast Data Control Plane**. + +It provides an overall **visualization** of the Fast Data pipelines released on a specific namespace, and offers functionalities to **pause and resume the consumption of data streams** along Fast Data pipelines with just a few clicks. + +![Control Plane Fast Data Pipeline](img/fast-data-pipeline.png) + +Fast Data Control Plane enables strategic management of the runtime environment in various scenarios: + +- **Visualize Fast Data Pipelines while building them**: + - During the Fast Data pipeline development phase, users can seamlessly configure and deploy new pipeline steps while the Control Plane UI provides a step-by-step visual representation of the architecture. + - Development is streamlined through incremental rendering on the frontend with continuous deployment, offering immediate visual feedback as the pipeline is built. + +- **Strategic resource allocation and performance optimization**: + - Dynamically allocate resources by pausing and resuming microservices involved in messages consumption based on workload and resource availability. + - Efficiently utilize computing resources and prioritize critical processes during peak demand. + +- **Initial Load and Full Refresh processes management**: + - Govern and control each step of the Initial Load (or Full Refresh) from the Control Plane UI without any frictions. + - Monitor consumer lag and workload health during these processes to ensure smooth execution. + +- **Performance Testing and Simulation**: + - Simulate different scenarios for performance testing by pausing and resuming messages consumption along pipeline. + - Observe system behavior under various conditions to identify bottlenecks and optimize resource allocation. + +- **Enhanced System Reliability**: + - Gracefully handle unexpected situations or system maintenance by pausing specific data pipeline steps. + - Resume operations post-maintenance to minimize downtime and enhance overall system reliability. + +## Architecture + +![Control Plane v2 Architecture](img/control-plane-v2-architecture.png) + +The Fast Data Control Plane architecture is designed to provide comprehensive monitoring and control capabilities for Fast Data pipelines deployed within a Kubernetes namespace. + +The Control Plane consists of two main services that must be deployed in the same namespace as the Fast Data Engine workloads: + +- **Control Plane Frontend**: Provides the user interface for visualizing and managing Fast Data pipelines. Users interact with this web-based interface to monitor pipeline status, pause/resume operations, and access real-time feedback about the system state. + +- **Control Plane**: Acts as the orchestration engine that communicates with the Fast Data Engine workloads. It manages state transitions, coordinates pause/resume operations across pipeline components, and ensures consistent pipeline governance. + +All components communicate through dedicated **State/Feedback channels** that enable: +- Real-time status monitoring and health checks +- Coordinated pause/resume operations across pipeline steps +- Bidirectional communication for control commands and status updates + +The **API Gateway** provides secure access to the Control Plane services, while **States Persistence** MongoDB collection ensures that pipeline states and configurations are maintained consistently across system restarts and deployments. + +## Useful Resources + +Here are some useful links to start adopting Runtime Management features into your Fast Data v2 solution: + +- visit the [Control Plane UI documentation](/products/fast_data_v2/runtime_management/control_plane_ui.md) to learn how to interact with the Control Plane frontend interface and manage your Fast Data pipelines; +- visit the [Application Configuration documentation](/products/fast_data_v2/runtime_management/application_configuration.md) to understand how to configure the Control Plane application and to enable the communication with the Fast Data Engine workloads; +- visit the [Compatibility Matrix](/products/fast_data_v2/runtime_management/compatibility_matrix.md) to check whether your infrastructure and Fast Data v2 services are compatible with the Runtime Management features; +- visit the [Best Practices documentation](/products/fast_data_v2/runtime_management/best_practices.md) for recommendations on initial load strategies, monitoring approaches, and optimization techniques. diff --git a/sidebars.json b/sidebars.json index aee6fa4c7a..6edbc595f8 100644 --- a/sidebars.json +++ b/sidebars.json @@ -2273,6 +2273,33 @@ "type": "doc" } ] + }, + { + "label": "Runtime Management", + "type": "category", + "collapsed": false, + "link": { + "id": "products/fast_data_v2/runtime_management/overview", + "type": "doc" + }, + "items": [ + { + "id": "products/fast_data_v2/runtime_management/control_plane_ui", + "type": "doc" + }, + { + "id": "products/fast_data_v2/runtime_management/application_configuration", + "type": "doc" + }, + { + "id": "products/fast_data_v2/runtime_management/compatibility_matrix", + "type": "doc" + }, + { + "id": "products/fast_data_v2/runtime_management/best_practices", + "type": "doc" + } + ] } ] }