Merge pull request #32 from dbazhenov/version_0.1.9

Release version 0.1.9 . Templates improvement. Reorganization of Dock…

Author: dbazhenov

README.md

@@ -46,10 +46,6 @@ The application connects to and generates load on MySQL, PostgreSQL, and MongoDB
 
    Open the folder with the repository `cd github-stat/`
 
-<!-- 2. Copy or rename `.env.example` (already provided in this repo) to `.env`. Set the parameters in the `.env` file.
-
-   > **Note:** The `.env` file contains essential configuration settings for the application. Adjust these settings based on your environment to ensure proper functionality. -->
-
 2. Run the environment. Two options:
 
 - Demo application only. Suitable for connecting to your own databases e.g. created with Percona Everest, Pecona Operators or other databases in the cloud or locally.
@@ -58,13 +54,13 @@ The application connects to and generates load on MySQL, PostgreSQL, and MongoDB
    docker compose up -d
    ```
 
-- Demo application with test databases (MySQL, MongoDB, Postgres) and PMM.
+- Demo application with test databases (MySQL 8.4, MongoDB 8, Postgres 17) and Percona Monitoring and Management (PMM).
 
    ```bash
-   docker compose -f docker-compose-full.yaml up -d
+   docker-compose -p demo-app -f docker/full.yaml up -d
    ```
 
-   > **Note:** We recommend looking at the docker-compose.yaml files so you can know which containers are running and with what settings. You can always change the settings.
+   > **Note:** We recommend looking at the docker-compose files so you can know which containers are running and with what settings. You can always change the settings.
 
    > **Note:** PMM server will be available at `localhost:8080`, access `admin` / `admin` . At the first startup, it will offer to change the password, skip it or set the same password (admin). 
 
@@ -80,8 +76,6 @@ The application connects to and generates load on MySQL, PostgreSQL, and MongoDB
 
    - **Postgres**: `user=postgres password='password' dbname=dataset host=postgres port=5432 sslmode=disable`
 
-   - **YugabyteDB**: `user=yugabyte password='password' dbname=dataset host=yugabytedb port=5433 sslmode=disable` (YugabyteDB UI is on port 15433)
-
    - **MongoDB**: `mongodb://databaseAdmin:password@mongodb:27017/`
 
    If you connect to your databases, you probably know the settings to connect, if not, write to us.
@@ -104,35 +98,55 @@ The application connects to and generates load on MySQL, PostgreSQL, and MongoDB
 
    > **Note:** You can see the queries running in the QAN section of PMM, and you can also see the source code in the internal/load files for each database type.
 
-## Development Environment
+### Additional databases 
 
-0. Run the environment:
+The application can work with other compatible databases such as YugabyteDB, FerretDB or MariaDB
 
-   ```bash
-   docker compose -f docker-compose-dev.yaml up -d
-   ```
+To start the databases use docker-compose files from the docker folder or instructions from official sites.
 
-1. Run the Control Panel script:
+Start the application if it is not already running
+```bash
+docker-compose -p demo-app -f docker/app.yaml up -d
+```
 
-   ```go
-   go run cmd/web/main.go
-   ```
+**YugabyteDB**
 
-   Launch the control panel at localhost:3000.
+Run docker compose with the YugabyteDB database
 
-2. Run the Dataset loader script
+```bash
+docker-compose -p demo-app -f docker/yugabytedb.yaml up -d
+```
 
-   ```go
-   go run cmd/dataset/main.go
-   ```
+Open the Settings tab in the Control Panel and create a connection
 
-3. Run the Dataset Loader script:
+```user=yugabyte password='password' dbname=dataset host=yugabytedb port=5433 sslmode=disable``` 
 
-   ```go
-   go run cmd/load/main.go
-   ```
+YugabyteDB UI is on port 15433
 
-   Start PMM in your browser at `localhost:8080` (admin/admin).
+**FerretDB**
+
+Run docker compose with the FerretDB database
+
+```bash
+docker-compose -p demo-app -f docker/ferretdb.yaml up -d
+```
+
+Open the Settings tab in the Control Panel and create a connection
+```
+mongodb://username:password@ferretdb/ferretdb?authMechanism=PLAIN
+```
+
+**MariaDB**
+
+Run docker compose with the MariaDB database
+
+```bash
+docker-compose -p demo-app -f docker/mariadb.yaml up -d
+```
+
+Open the Settings tab in the Control Panel and create a connection
+
+```root:password@tcp(mariadb:3306)/dataset```
 
 ## Launching in Kubernetes
 
@@ -283,6 +297,74 @@ The first time you connect to MySQL and Postgres, you will need to create a sche
 
 Have fun experimenting.
 
+## Development Environment
+
+0. Run the environment:
+
+   ```bash
+   docker compose -p demo-app -f docker/dev.yaml up -d
+   ```
+
+   This will start the Valkey required for the application services and the three databases (MySQL 8.4, MongoDB 8, Postgres 17). Edit docker/dev.yaml if you need other databases or versions.
+
+1. Run the Control Panel script:
+
+   ```go
+   go run cmd/web/main.go
+   ```
+
+   Launch the control panel at localhost:3000. Open the Settings tab and add connections. The control panel is a web application, the settings are saved in Valkey. 
+
+2. Run the Dataset loader script
+
+   ```go
+   go run cmd/dataset/main.go
+   ```
+
+   This will start the load service. The service reads the configuration from Valkey according to the control panel and generates the load in separate Go routines.
+
+3. Run the Dataset Loader script:
+
+   ```go
+   go run cmd/load/main.go
+   ```
+
+   Start PMM in your browser at `localhost:8080` (admin/admin).
+
+## Release process of the new version
+
+1. Test the application in a dev environment. Check the logs in the console.
+
+2. Change the image versions for demo_app_dataset, demo_app_load, demo_app_web to the new version number in the files:
+
+   *  `docker-compose.yaml`
+
+   *  `docker/app.yaml`
+
+   *  `docker/full.yaml`
+
+   *  `k8s/helm/Chart.yaml`
+
+   *  `k8s/helm/values.yaml` - images section
+
+   *  `k8s/manual/*` - dataset-deployment.yaml, web-deployment.yaml, load-deployment.yaml files
+
+3. Building and publishing to docker hub is done by GitHub Workflow by tag automatically. Set a new tag with the command:
+
+   ```
+   git tag -a 0.1.9 -m "Release 0.1.9"
+   ``` 
+
+   Publish a new tag
+
+   ``` 
+   git push origin 0.1.9
+   ``` 
+
+4. Check that the GitHub Action is successful and new versions are published on dockerhub. 
+
+5. Test the application in docker and k8s
+
 ### Useful Commands
 
 - Get Pods:

cmd/load/main.go

@@ -346,7 +346,7 @@ func runMySQL(ctx context.Context, routineId int, dbConfig map[string]string) {
 			return
 		default:
 			// Update localDBConfig every 2 seconds
-			if time.Since(lastUpdate) > 2*time.Second {
+			if time.Since(lastUpdate) > 1*time.Second {
 				localDBConfig = getDatabaseByID(localDBConfig["id"], "mysql")
 				if localDBConfig == nil {
 					log.Printf("MySQL: %s: goroutine: %d: database has been removed, stopping goroutine", dbConfig["id"], routineId)
@@ -415,7 +415,7 @@ func runPostgreSQL(ctx context.Context, routineId int, dbConfig map[string]strin
 			return
 		default:
 
-			if time.Since(lastUpdate) > 2*time.Second {
+			if time.Since(lastUpdate) > 1*time.Second {
 				localDBConfig = getDatabaseByID(dbConfig["id"], "postgres")
 
 				if localDBConfig == nil {
@@ -487,7 +487,7 @@ func runMongoDB(ctx context.Context, routineId int, dbConfig map[string]string)
 			return
 		default:
 
-			if time.Since(lastUpdate) > 2*time.Second {
+			if time.Since(lastUpdate) > 1*time.Second {
 				localDBConfig = getDatabaseByID(dbConfig["id"], "mongodb")
 
 				if localDBConfig == nil {

docker-compose.yaml

@@ -15,7 +15,7 @@ services:
       retries: 5
 
   demo_app_dataset:
-    image: dbazhenov/demo_app_dataset:0.1.8
+    image: dbazhenov/demo_app_dataset:0.1.9
     depends_on:
       valkey:
         condition: service_healthy
@@ -31,7 +31,7 @@ services:
       - DEBUG=false
 
   demo_app_web:
-    image: dbazhenov/demo_app_web:0.1.8
+    image: dbazhenov/demo_app_web:0.1.9
     depends_on:
       valkey:
         condition: service_healthy
@@ -43,7 +43,7 @@ services:
       - "3000:3000"
 
   demo_app_load:
-    image: dbazhenov/demo_app_load:0.1.8
+    image: dbazhenov/demo_app_load:0.1.9
     depends_on:
       valkey:
         condition: service_healthy

docker/app.yaml

@@ -0,0 +1,58 @@
+version: '3.8'
+services:
+  valkey:
+    image: valkey/valkey:8
+    environment:
+      - ALLOW_EMPTY_PASSWORD=yes
+    ports:
+      - "6379:6379"
+    volumes:
+      - valkey-data:/data
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  demo_app_dataset:
+    image: dbazhenov/demo_app_dataset:0.1.9
+    depends_on:
+      valkey:
+        condition: service_healthy
+    environment:
+      - VALKEY_ADDR=valkey 
+      - VALKEY_PORT=6379
+      - DATASET_LOAD_TYPE=csv # github or csv
+      - DATASET_DEMO_CSV_PULLS=data/csv/pulls.csv # https://github.com/dbazhenov/github-stat/raw/refs/heads/main/data/csv/pulls.csv.zip
+      - DATASET_DEMO_CSV_REPOS=data/csv/repositories.csv # https://github.com/dbazhenov/github-stat/raw/refs/heads/main/data/csv/repositories.csv.zip
+      - GITHUB_ORG=percona # required for github load type
+      - GITHUB_TOKEN= # required for github load type
+      - DELAY_MINUTES=10
+      - DEBUG=false
+
+  demo_app_web:
+    image: dbazhenov/demo_app_web:0.1.9
+    depends_on:
+      valkey:
+        condition: service_healthy
+    environment:
+      - VALKEY_ADDR=valkey 
+      - VALKEY_PORT=6379
+      - CONTROL_PANEL_PORT=3000
+    ports:
+      - "3000:3000"
+
+  demo_app_load:
+    image: dbazhenov/demo_app_load:0.1.9
+    depends_on:
+      valkey:
+        condition: service_healthy
+    environment:
+      - VALKEY_ADDR=valkey
+      - VALKEY_PORT=6379
+      - LOAD_MYSQL=true
+      - LOAD_POSTGRES=true
+      - LOAD_MONGODB=true
+
+volumes:
+  valkey-data:

docker-compose-dev.yaml docker/dev.yamldocker-compose-dev.yaml renamed to docker/dev.yaml

@@ -1,3 +1,4 @@
+version: '3.8'
 services:
 
   valkey:
@@ -7,23 +8,22 @@ services:
     ports:
       - "6379:6379"
     volumes:
-      - valkey_data:/data
+      - valkey-data:/data
     healthcheck:
       test: ["CMD", "redis-cli", "ping"]
       interval: 5s
       timeout: 3s
       retries: 5
 
   postgres:
-    image: "percona/percona-distribution-postgresql:16.2-multi"
+    image: "percona/percona-distribution-postgresql:17-multi"
     environment:
       POSTGRES_PASSWORD: password
       POSTGRES_USER: postgres
       LANG: en_US.utf8
       PGDATA: /data/db
     volumes:
-      - pgdata:/data/db
-      - ./data/init/postgresql/init.sql:/docker-entrypoint-initdb.d/init.sql
+      - pg-data:/data/db
     ports: 
       - "5432:5432"
     healthcheck:
@@ -32,32 +32,13 @@ services:
       timeout: 10s
       retries: 5
 
-  yugabytedb:
-    image: "yugabytedb/yugabyte:2024.2.0.0-b145"
-    command: yugabyted start --enable_pg_parity_early_access --tserver_flags="ysql_pg_conf_csv={shared_preload_libraries='pg_stat_monitor'}" --initial_scripts_dir=/docker-entrypoint-initial_scripts_dir --background=false 
-    environment:
-      YSQL_PASSWORD: password
-      YSQL_USER: yugabyte
-    volumes:
-       - ./data/init/postgresql:/docker-entrypoint-initial_scripts_dir
-    ports: 
-      - "5433:5433"
-      - "15433:15433"
-    healthcheck:
-      test: ["CMD-SHELL", "PGPASSWORD=$$YSQL_PASSWORD yugabyted connect ysql <<<''"]
-      interval: 30s
-      timeout: 10s
-      retries: 5
-
   mongodb:
-    image: "percona/percona-server-mongodb:7.0-multi"
+    image: "percona/percona-server-mongodb:8.0-multi"
     volumes:
-      - mongodata:/data/db
-      - ./data/init/mongodb/init.js:/docker-entrypoint-initdb.d/init.js:ro
+      - mongo-data:/data/db
     environment:
       MONGO_INITDB_ROOT_USERNAME: databaseAdmin
       MONGO_INITDB_ROOT_PASSWORD: password
-      MONGO_INITDB_DATABASE: dataset
     ports:
       - "27017:27017"
     healthcheck:
@@ -67,13 +48,11 @@ services:
       retries: 5
 
   mysql:
-    image: "percona/percona-server:8.3.0-1.1-multi"
+    image: "percona/percona-server:8.4.3-3.1-multi"
     volumes:
-      - mysqldata:/var/lib/mysql
-      - ./data/init/mysql:/docker-entrypoint-initdb.d
+      - mysql-data:/var/lib/mysql
     environment:
       MYSQL_ROOT_PASSWORD: password
-      MYSQL_DATABASE: dataset
     healthcheck:
       test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-uroot", "-proot-password"]
       interval: 5s
@@ -131,8 +110,8 @@ services:
         pmm-admin add mongodb --username=databaseAdmin --password=password --host=mongodb --port=27017 --query-source=profiler
 
 volumes:
+  valkey-data:
   pmm-data:
-  mongodata:
-  pgdata:
-  mysqldata:
-  valkey_data:
+  mongo-data:
+  pg-data:
+  mysql-data: