diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/control_plane_ui.md b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/control_plane_ui.md index 4f5c824eb3..469a357b1c 100644 --- a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/control_plane_ui.md +++ b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/control_plane_ui.md @@ -7,21 +7,15 @@ sidebar_label: Control Plane UI The Fast Data Control Plane UI provides a comprehensive visual interface for managing and monitoring your Fast Data v2 pipelines. This web-based interface allows you to visualize the entire data pipeline architecture, monitor real-time performance, and control runtime states of your Fast Data Engine workloads with just a few clicks. -## Artifacts - Fast Data Control Plane shows various artifacts that represent different aspects of the overall architecture of Fast Data pipelines. -### Data Streams +## Data Streams Data Streams represent the channels through which data flows between different execution steps in your pipeline. In the Fast Data v2 architecture, these are Kafka topics that typically carry [Fast Data compliant messages](/products/fast_data_v2/concepts.mdx#fast-data-message-format) between workloads. -Each Data Stream lists its associated consumers, detailing essential metadata such as consumer names, runtime states, and the consumer groups that define each Fast Data workload. - -### Persisted Assets +![Data Stream Detail](img/data-stream-detail.png) -Persisted Assets represent data that is stored persistently at a certain step of the pipeline. These can be often the final data products (Single Views) that result from your Fast Data pipeline operations. - -Each Persisted Asset displays useful metadat and other information about the persistor. +Each Data Stream lists its associated consumers, detailing essential metadata such as consumer names, runtime states, and the consumer groups that define each Fast Data workload. ## Execution Steps @@ -31,6 +25,10 @@ Execution Steps are the core processing components of your Fast Data pipeline, i The **Ingest** step is implemented by [Mongezium](/products/fast_data_v2/mongezium_cdc/10_Overview.md), that acts as Change Data Capture (CDC) from MongoDB collections to Kafka streams. +:::note +If input data streams originate from a custom CDC plugin or a CDC deployed in a different Kubernetes namespace, the Ingest execution step will not be rendered in Control Plane canvas. Consequently, visibility is limited to input streams consumed by Fast Data workloads within the current namespace. +::: + The Ingest step monitors MongoDB change streams in real-time and converts database operations (insert, update, delete) into Fast Data compliant messages that flow to downstream processing steps. When you click on an Ingest step, the detail panel shows two tabs: @@ -38,9 +36,13 @@ When you click on an Ingest step, the detail panel shows two tabs: - **Output Topics**: Lists all Kafka topics where change events are published. Each topic corresponds to a MongoDB collection being monitored for changes. - **Details**: Provides useful metadata like the type of the execution step, the version of the microservice that implements the steps and the number of replicas. +![Ingest Detail](img/ingest-detail.png) + ### Process -The **Process** step is implemented by [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md) and provides powerful data transformation capabilities using custom JavaScript logic executed in a secure sandbox environment. +The **Process** step is implemented by [Stream Processor](/products/fast_data_v2/stream_processor/10_Overview.md), that provides powerful data transformation capabilities. + +![Process Detail](img/process-detail.png) The Process step detail panel contains three tabs: @@ -50,7 +52,9 @@ The Process step detail panel contains three tabs: ### Aggregate -The **Aggregate** step is implemented by [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) and performs real-time multi-stream data aggregation to create data products. +The **Aggregate** step is implemented by [Farm Data](/products/fast_data_v2/farm_data/10_Overview.md) and performs real-time multi-stream data aggregation useful to create data products. + +![Aggregate Detail](img/aggregate-detail.png) The Aggregate step offers three tabs in its detail panel: @@ -62,6 +66,8 @@ The Aggregate step offers three tabs in its detail panel: The Aggregation Graph Canvas is a specialized visual interface within the Aggregate step that displays the entity relationship diagram configured for your aggregation logic. This canvas shows how different data streams are structurally combined to produce the final aggregated output. +![Aggregation Graph Detail](img/aggregation-graph-detail.png) + The Aggregation Graph Canvas offers a comprehensive visual representation of entity relationships within the aggregation process, clearly illustrating how different data entities are related with each other to build the final aggregated output. This interface provides detailed information about relationship configurations that define how entities are joined and aggregated together. The canvas enables interactive exploration of the entire aggregation logic, allowing you to select any two data streams in the graph to examine the specific relationship configuration that governs their interaction, including join conditions and aggregation rules. @@ -71,6 +77,8 @@ Moreover, from the Aggregation Graph Canvas, it is possible to control the runti The **Persist** step is implemented by [Kango](/products/fast_data_v2/kango/10_Overview.md) and handles the final storage of processed data from Kafka streams to MongoDB collections (Persisted Assets). +![Persist Detail](img/persist-detail.png) + The Persist step detail panel includes two tabs: - **Consumers**: Shows the input stream that Kango is consuming for persistence, including consumer group details and consumer lag metrics. @@ -82,9 +90,11 @@ Each execution step in your pipeline can be in one of four distinct runtime stat ### Running -The **Running** state indicates that the workload's consumer is actively consuming data from the input stream. +The **Running** state indicates that the workload's consumer is actively consuming data from the input stream. + +![Running State](img/running-state.png) -Running steps are typically indicated by blue dotted line in the UI. +Running steps are indicated by blue dotted line in the UI. ### Paused @@ -92,7 +102,9 @@ The **Paused** state means that the workload's consumer has stopped consuming da In Paused state, the workload remains healthy and responsive, no new messages are being consumed from input streams; thus, in case of new input messages, the displayed consumer lag will consequently increase. -Paused steps are typically indicated by grey line in the UI. +![Pause State](img/pause-state.png) + +Paused steps are indicated by grey line in the UI. ### Unsubscribed @@ -100,6 +112,8 @@ The **Unsubscribed** state indicates that the workload's consumer is not subscri This commonly occurs when the consumer group has been unsubscribed from that topic, or when the workload has no replicas running (scaled down completely). +![Unsubscribed State](img/unsubscribed-state.png) + In this state, the consumer cannot process any data and it is no more possible to interact with Play/Pause button and to know about consumer lag info. Unsubscribed steps are typically indicated by orange line in the UI. @@ -110,7 +124,9 @@ The **Unknown** state indicates that the Control Plane cannot determine the curr Unknown states require investigation and usually indicate operational issues that need immediate attention. -Unknown steps are typically indicated by red line in the UI. +![Unknown State](img/unknown-state.png) + +Unknown steps are indicated by red line in the UI. ## Control Runtime States @@ -118,6 +134,8 @@ The Control Plane UI provides powerful controls for managing the runtime state o Each data flows inside the pipeline provides **Pause** and **Play** buttons that allow you to stop and start data consumption from the data stream. +![Button](img/button.png) + - **Pause**: Stops the consumer from processing new messages while maintaining stream position - **Resume**: Restarts data consumption from the previously maintained position @@ -132,7 +150,9 @@ The Control Plane UI provides several features to help you efficiently navigate ### Search elements on the canvas -The search feature enables you to quickly locate specific elements within your pipeline. +The search feature enables you to quickly locate specific elements within your pipeline. + +![Search List](img/search-list.png) Simply type on the searchbar the name of an artifact or Fast Data workload and select it. The interface will instantly focus on and highlight the selected element, automatically opening the contextual drawer to display all relevant information about it. @@ -140,4 +160,6 @@ Simply type on the searchbar the name of an artifact or Fast Data workload and s Typically, when you hover over elements listed in the detail panels or drawers, the corresponding involved pipeline elements are instantly highlighted inside the canvas. +![Highlight Detail](img/highlight-detail.png) + This highlighting feature significantly improves navigation efficiency, especially in complex pipelines with many interconnected components. diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregate-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregate-detail.png new file mode 100644 index 0000000000..5ccff39ead Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregate-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregation-graph-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregation-graph-detail.png new file mode 100644 index 0000000000..9dfc03aacf Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/aggregation-graph-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/button.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/button.png new file mode 100644 index 0000000000..27d648c8a0 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/button.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/data-stream-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/data-stream-detail.png new file mode 100644 index 0000000000..ff8eb62e4f Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/data-stream-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/highlight-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/highlight-detail.png new file mode 100644 index 0000000000..d9a6107bd0 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/highlight-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/ingest-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/ingest-detail.png new file mode 100644 index 0000000000..98eb8371db Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/ingest-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/pause-state.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/pause-state.png new file mode 100644 index 0000000000..5e4cf2bd65 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/pause-state.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/persist-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/persist-detail.png new file mode 100644 index 0000000000..2bafe95da2 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/persist-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/process-detail.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/process-detail.png new file mode 100644 index 0000000000..9bb5932ecc Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/process-detail.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/running-state.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/running-state.png new file mode 100644 index 0000000000..b011bb8b24 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/running-state.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/search-list.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/search-list.png new file mode 100644 index 0000000000..4c5a6c51a7 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/search-list.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unknown-state.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unknown-state.png new file mode 100644 index 0000000000..49d3c3119b Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unknown-state.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unsubscribed-state.png b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unsubscribed-state.png new file mode 100644 index 0000000000..429fac5c01 Binary files /dev/null and b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/img/unsubscribed-state.png differ diff --git a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/overview.md b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/overview.md index 036c2de9d2..93ac6bd31b 100644 --- a/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/overview.md +++ b/versioned_docs/version-14.5.0/products/fast_data_v2/runtime_management/overview.md @@ -17,7 +17,7 @@ It provides an overall **visualization** of the Fast Data pipelines released on Fast Data Control Plane enables strategic management of the runtime environment in various scenarios: - **Visualize Fast Data Pipelines while building them**: - - During the Fast Data pipeline development phase, users can seamlessly configure and deploy new pipeline steps while the Control Plane UI provides a step-by-step visual representation of the architecture. + - During the Fast Data pipeline development phase, users can seamlessly configure Fast Data Engine workloads in Design Area and deploy new data pipeline steps while the Control Plane UI provides a step-by-step visual representation of the architecture increments. - Development is streamlined through incremental rendering on the frontend with continuous deployment, offering immediate visual feedback as the pipeline is built. - **Strategic resource allocation and performance optimization**: @@ -26,7 +26,7 @@ Fast Data Control Plane enables strategic management of the runtime environment - **Initial Load and Full Refresh processes management**: - Govern and control each step of the Initial Load (or Full Refresh) from the Control Plane UI without any frictions. - - Monitor consumer lag and workload health during these processes to ensure smooth execution. + - Monitor consumer lag and workload runtime state during these processes to ensure smooth execution. - **Performance Testing and Simulation**: - Simulate different scenarios for performance testing by pausing and resuming messages consumption along pipeline. @@ -46,14 +46,14 @@ The Control Plane consists of two main services that must be deployed in the sam - **Control Plane Frontend**: Provides the user interface for visualizing and managing Fast Data pipelines. Users interact with this web-based interface to monitor pipeline status, pause/resume operations, and access real-time feedback about the system state. -- **Control Plane**: Acts as the orchestration engine that communicates with the Fast Data Engine workloads. It manages state transitions, coordinates pause/resume operations across pipeline components, and ensures consistent pipeline governance. +- **Control Plane**: Acts as the orchestration engine that communicates with the Fast Data workloads. It manages state transitions, coordinates pause/resume operations across pipeline components, and ensures consistent pipeline governance. All components communicate through dedicated **State/Feedback channels** that enable: - Real-time status monitoring and health checks - Coordinated pause/resume operations across pipeline steps -- Bidirectional communication for control commands and status updates +- Communication for control commands and runtime state updates -The **API Gateway** provides secure access to the Control Plane services, while **States Persistence** MongoDB collection ensures that pipeline states and configurations are maintained consistently across system restarts and deployments. +The **Runtime States Persistence** MongoDB collection ensures that pipeline states and configurations are maintained consistently across system restarts and incremental deployments. ## Useful Resources