dp-support

This repo is part of the Data Platform project. The Data Platform consists of services for capturing and providing access to data captured from a particle accelerator facility. The data-platform repo provides a project overview and links to the various project componnents, as well as an installer for running the latest version.

This repo, dp-support includes a set of utilities for managing the processes comprising the Data Platform ecosystem.

The Data Platform ecosystem consists of the following components:

• the service implementations in the dp-service repo
• MongoDB, used for persistence by the Data Platform
• (optionally) An Envoy proxy, for sending http1 traffic from the Data Platform web application to the http2-based gRPC service implementations
• (optionally) An Apache web server, for serving files generated by the export operation.

The dp-support repo includes tools for managing each of these components. Use of the tools is optional. They can be used as is (by downloading the installer from the data-platform repo), or can be treated as a starting place for a custom installation.

Note that Docker installation with command-line access to "docker" is a prerequisite to use the docker-based scripts. Docker is not required for a production installation, but can be a useful way to deploy demo, test, development, and QA systems.

The utilities for each component are described in more detail below.

Quick Start

The data-platform repo includes a Quick Start section for quickly getting the Data Platform up and running, with example usage of the scripts in dp-support to manage the ecosystem.

MongoDB scripts

MongoDB is used for Data Platform persistence. The current reference version is 8.0.6. For details about installing MongoDB as a locally installed package or as a docker container deployment, see the data-platform repo documentation. The tools for managing MongoDB include:

local MongoDB installation

One set of scripts is used for managing a Linux MongoDB installation using the "systemctl" utility. These are simple wrapper scripts:

• mongodb-systemctl-start: Starts standard MongoDB installation using systemctl.
• mongodb-systemctl-stop: Stops MongoDB.
• mongodb-systemctl-status: Checks MongoDB status.
• mongodb-systemctl-enable: Enables MongoDB auto-start after reboot.
• mongodb-systemctl-disable: Disables MongoDB auto-start after reboot.

Docker MongoDB deployment

The second set of scripts is for managing MongoDB deployed as a Docker container. To use these scripts, docker must be installed on the host with command-line access to the "docker" command. The following scripts are included:

• mongodb-docker-create: Creates MongoDB Docker container.
• mongodb-docker-start: Starts the MongoDB Docker container.
• mongodb-docker-stop: Stops the MongoDB Docker container.
• mongodb-docker-remove: Removes the MongoDB Docker container. BEWARE THIS WILL DELETE ALL DATA IN THE DATABASE.
• mongodb-docker-shell: Runs mongosh shell against the (running) MongoDB Docker container.

MongoDB Compass GUI application

Compass is a GUI application for navigating a MongoDB database. This repo contains a simple wrapper script for starting the application, passing the default MongoDB connection string (MongoDB user/password = "admin") on start up:

mongodb://admin:admin@localhost:27017

This script will need to be modified as appropriate if the user/password are customized.

• mongodb-compass-start: Starts the MongoDB Compass application, passing the default connection string on the command line.

Envoy proxy scripts

The Envoy proxy is an optional part of the Data Platform ecosystem. It is needed to run the web application, for converting the http1 traffic from the JavaScript browser application into http2 for interacting with the Data Platform services.

Scripts are provided to support deployment of the Envoy proxy as a Docker container, including:

• envoy-docker-create: Creates Envoy Docker container.
• envoy-docker-start: Starts the Envoy Docker container.
• envoy-docker-stop: Stops the Envoy Docker container.
• envoy-docker-remove: Removes the Envoy Docker container.

These scripts use config files "envoy.yaml" and "envoy.mac.yaml" in the "dp-support/config" directory.

Apache docker scripts

The Apache web server is an optional part of the Data Platform ecosystem. It can be used to provide access to files generated by the export operation.

Scripts are provided to support deployment of the Apache server as a Docker container, including:

• apache-docker-create: Creates Apache Docker container.
• apache-docker-start: Starts the Apache Docker container.
• apache-docker-stop: Stops the Apache Docker container.
• apache-docker-remove: Removes the Apache Docker container.

Data platform server scripts

The bin directory contains scripts for managing the Data Platform server applications, including the Ingestion and Query Services. These scripts use a set of lower level scripts in the same directory for starting/stopping processes and checking their status: util-pm-start, util-pm-stop, and util-pm-status, respectively.

Ingestion Service scripts

• server-ingest-start: Starts the ingestion server application using the util-pm-start script.
• server-ingest-stop: Stops the running ingestion server application using the util-pm-stop script.
• server-ingest-status: Checks the status of the ingestion server application using util-pm-status.

Query Service scripts

• server-query-start: Starts the query server application using the util-pm-start script.
• server-query-stop: Stops the running query server application using the util-pm-stop script.
• server-query-status: Checks the status of the query server application using util-pm-status.

Annotation Service scripts

• server-annotation-start: Starts the annotation server application using the util-pm-start script.
• server-annotation-stop: Stops the running annotation server application using the util-pm-stop script.
• server-annotation-status: Checks the status of the annotation server application using util-pm-status.

Ingestion Stream Service scripts

• server-ingestion-stream-start: Starts the ingestion stream server application using the util-pm-start script.
• server-ingestion-stream-stop: Stops the running ingestion stream server application using the util-pm-stop script.
• server-ingestion-stream-status: Checks the status of the ingestion stream server application using util-pm-status.

GUI application scripts

The bin directory contains scripts for running Data Platform gui applications.

• app-run-desktop-app: Starts the desktop GUI application.

Data platform performance benchmarks

The Data Platform includes applications for running Ingestion and Query Service performance benchmarks. The benchmarks require running non-standard Ingestion and Query servers that override the default port and database name, and then a client application for running scenarios and measuring performance.

The bin directory includes bash scripts for running the benchmark server and client applications.

Ingestion Service benchmark scripts

ingestion benchmark server scripts

• server-benchmark-ingest-start: Starts the benchmark ingestion server application using the util-pm-start script.
• server-benchmark-ingest-stop: Stops the running benchmark ingestion server application using the util-pm-stop script.
• server-benchmark-ingest-status: Checks the status of the benchmark ingestion server application using util-pm-status.

ingestion benchmark application scripts

• app-run-ingestion-benchmark: Runs the data platform ingestion service performance benchmark application against the running benchmark ingestion server. Displays an error if the server is not running. Uploads one minute's data for 4,000 data sources sampled at 1 KHz.
• app-run-ingestion-bytes-benchmark: Runs the same ingestion scenario as the regular ingestion benchmark, but uses the byte data mechanism for sending SerializedDataColumns instead of regular DataColumns in the ingestion requests.

Query Service benchmark scripts

query benchmark server scripts

• server-benchmark-query-start: Starts the benchmark query server application using the util-pm-start script.
• server-benchmark-query-stop: Stops the running benchmark query server application using the util-pm-stop script.
• server-benchmark-query-status: Checks the status of the benchmark query server application using util-pm-status.

query benchmark application scripts

• app-run-query-benchmark: Runs the query service performance benchmark application using the server-streaming time-series data query API. Loads a set of data and exercises the query mechanism against it.
• app-run-query-bytes-benchmark: Runs the same query scenario as the regular query benchmark, but uses the byte data mechanism for receiving SerializedDataColumns instead of regular DataColumns in the query result.

Sample data generator scripts

The Data Platform includes a utility for generating sample data for use in web application development and demo purposes. The application generates sample data for one minute and 4,000 signals each sampled at 1 KHz. The data generator uses the standard "dp" database and data is created with a fixed date/time starting at "2023-10-31T15:51:00.000+00:00". Happy Halloween...

• app-run-test-data-generator: Runs the sample data generator against the running ingesiton server application.