[QS220014] CGA deploy

README.md

@@ -1,10 +1,8 @@
-# Law Enforcement Media Application (LEMA)
-
-The following installation steps compliment the full LEMA Administration Guide, available from the [IDOL documentation site](https://www.microfocus.com/documentation/idol/), under "IDOL Government Solutions".
+# Concept Graph Analytics (CGA)
 
 ## Basic deployment
 
-To deploy LEMA to a Docker system, use the `deploy.py` tool, which requires Docker Compose.
+To deploy CGA to a Docker system, use the `deploy.py` tool, which requires Docker Compose.
 Required software versions:
 - Python 3, version 3.6 or later
 - Docker, version 20.10.22 or later
@@ -27,13 +25,11 @@ Run the `deploy.py` tool using Python.  (Much like when running `docker`, you
 may have to run it as a different user with sufficient permissions to manage Docker containers.)
 
 ```
-python3 deploy.py --init auth entity filestore analysis audit api ui
+python3 deploy.py --init auth entity audit cga
 ```
 
-With the default configuration, the LEMA UI will be available at `https://localhost:8070` once the
-system has started.
-
-After the system has started, log in with a user that has the `admin` role. The LEMA UI will then perform a one off initialization. 
+After the system has started, log into the Swagger UI `localhost:8060/swagger/` with a user that has the `admin` role
+and call the POST `/meta/initialize` endpoint to perform a one off initialization.
 
 To show options and other usage information, run:
 
@@ -50,35 +46,33 @@ of the settings.
 
 ## Encryption
 
-By default, the user-facing servers (authentication server, API, and UI) only accept encrypted 
+By default, the user-facing servers (authentication server and API) only accept encrypted 
 connections.  For this to work, you must obtain TLS certificates and copy them into the `config` 
 directory.  The required files are:
 
 - `config/https/api/tls.key`: Private key for the API.
 - `config/https/api/tls.crt`: Server certificate for the API.
 - `config/https/auth/tls.key`: Private key for the authentication server.
 - `config/https/auth/tls.crt`: Server certificate for the authentication server.
-- `config/ui/auth/nginx.key`: Private key for the UI.
-- `config/ui/auth/nginx.crt`: Server certificate for the UI.
 
 ## Further examples
 
 To use HTTP instead of HTTPS, for testing purposes only, run:
 
 ```
-python3 deploy.py --disable-encryption --init auth entity filestore analysis analysis-live audit api ui
+python3 deploy.py --disable-encryption --init auth entity audit cga
 ```
 
 > note: changes to the encryption state of a deployed system require manual deletion of the realm in Keycloak before running `deploy.py` with the new state.
 
-To resume a stopped LEMA system, or to apply changes made to configuration files, or to change which
+To resume a stopped CGA system, or to apply changes made to configuration files, or to change which
 components are deployed: run the normal command to deploy, but without the `--init` argument:
 
 ```
-python3 deploy.py auth entity filestore analysis analysis-live audit api ui
+python3 deploy.py auth entity audit cga
 ```
 
-To stop and remove deployed LEMA services, run the Python `deploy.py` tool with no arguments:
+To stop and remove deployed CGA services, run the Python `deploy.py` tool with no arguments:
 
 ```
 python3 deploy.py
@@ -92,7 +86,7 @@ configure hosts and ports in the files in `config/`, and then run on separate ho
 
 ```
 python3 deploy.py audit
-python3 deploy.py --init auth entity analysis analysis-live api ui
+python3 deploy.py --init auth entity cga
 ```
 
 ## System information
@@ -107,14 +101,10 @@ while others listen on 127.0.0.1 only):
 | entity        | 8021     | no         | ACI port of IDOL Content database backend for the Gremlin database   |
 | entity        | 8022     | no         | Index port of IDOL Content database backend for the Gremlin database |
 | entity        | 8023     | no         | Port of Cassandra database backend for the Gremlin database          |
-| filestore     | 8030     | no         | S3-compatible object storage                                         |
-| analysis      | 8040     | no         | NiFi server for media analysis (API and admin UI)                    |
-| analysis-live | 8080     | no         | ACI port of IDOL Media Server used for live media analysis           |
 | audit         | 8050     | no         | PostgreSQL database storing audit logs                               |
 | api           | 8060     | yes        | System HTTP API                                                      |
-| ui            | 8070     | yes        | LEMA UI                                                              |
 
-Docker volumes are created with the prefix `micro-focus-idol-lema_`, which can be changed using the
+Docker volumes are created with the prefix `micro-focus-idol-cga_`, which can be changed using the
 `COMPOSE_PROJECT_NAME` setting.  The following volumes are created:
 
 | **Component** | **Volume name**                        | **Purpose**                                      |
@@ -123,10 +113,7 @@ Docker volumes are created with the prefix `micro-focus-idol-lema_`, which can b
 | entity        | entity-storagedb-data                  | Application data                                 |
 | entity        | entity-indexdb-data                    | Search index for application data                |
 | entity        | entity-indexdb-license-data            | Cache for license information                    |
-| filestore     | filestore-service-data                 | Uploaded and generated files                     |
-| analysis-live | analysis-live-mediaserver-license-data | Cache for license information                    |
-| analysis-live | analysis-live-mediaserver-output       | Temporary video and data files for live analysis |
 | audit         | audit-db-data                          | Audit logs                                       |
 
-All containers connect to a Docker network called `micro-focus-idol-lema_main`.  The
-`micro-focus-idol-lema` prefix can be changed using the `COMPOSE_PROJECT_NAME` setting.
+All containers connect to a Docker network called `micro-focus-idol-cga_main`.  The
+`micro-focus-idol-cga` prefix can be changed using the `COMPOSE_PROJECT_NAME` setting.

config-fixed/base.env

@@ -4,14 +4,10 @@ ISOL_DOCKER_NAME_SEP=/
 ISOL_DOCKER_VERSION_SEP=:
 ISOL_BASE_VERSION=2.3.1
 ISOL_API_VERSION=2.3.1
-ISOL_ANALYSIS_NIFI_VERSION=2.3.1
-ISOL_ANALYSIS_LIVE_VERSION=2.3.1
-LEMA_VERSION=2.3.1
 ISOL_POSTGRES_VERSION=14.4
 
 ISOL_EXTERNAL_PROTOCOL=https
 ISOL_ENTITY_SERVICE_TRAVERSAL_SOURCE=entitiesTraversal
 ISOL_AUTH_SERVICE_INTERNAL_PORT=8443
 ISOL_AUTH_SERVICE_TLS_REQUIRED=all
 ISOL_API_SERVICE_INTERNAL_PORT=8443
-LEMA_UI_INTERNAL_PORT=443

config-fixed/api.env → config-fixed/cga.env

@@ -1,6 +1,6 @@
-ISOL_API_SERVICE_APPLICATION_TYPE=lema
+ISOL_API_SERVICE_APPLICATION_TYPE=conceptgraph
 # relative to docker-compose/
-ISOL_API_SERVICE_TLS_CERT_DIR=../config/https/api
+ISOL_API_SERVICE_TLS_CERT_DIR=../config/https/cga
 ISOL_API_SERVICE_WATCHLIST_MIN_EVENT_AGE_SECONDS=60
 ISOL_API_SERVICE_STATS_MIN_ANALYSIS_JOB_EVENT_AGE_SECONDS=60
 ISOL_API_SERVICE_ADDITIONAL_ALLOWED_ORIGINS=

config-fixed/unencrypted.env

@@ -6,6 +6,3 @@ ISOL_AUTH_SERVICE_TLS_CERT_DIR=../config/empty
 
 ISOL_API_SERVICE_INTERNAL_PORT=8080
 ISOL_API_SERVICE_TLS_CERT_DIR=../config/empty
-
-LEMA_UI_INTERNAL_PORT=80
-LEMA_UI_TLS_CERT_DIR=../config/empty

config/auth.env

@@ -1,10 +1,10 @@
 # address on the Docker host to expose the configuration database port on
 ISOL_AUTH_DB_DOCKER_PORT=127.0.0.1:8000
 # credentials to configure and use for accessing the configuration database
-ISOL_AUTH_DB_USERNAME=lema
-ISOL_AUTH_DB_PASSWORD=lema
+ISOL_AUTH_DB_USERNAME=cga
+ISOL_AUTH_DB_PASSWORD=cga
 # name of the database to use for storing configuration in
-ISOL_AUTH_DB_NAME=lema-auth
+ISOL_AUTH_DB_NAME=cga-auth
 
 # address on the Docker host to expose the authentication server and management UI port on
 ISOL_AUTH_SERVICE_DOCKER_PORT=8010

config/base.env

@@ -1,5 +1,5 @@
 # prefix used for containers, volumes and networks
-COMPOSE_PROJECT_NAME=micro-focus-idol-lema
+COMPOSE_PROJECT_NAME=micro-focus-idol-cga
 
 # location of the License Server
 ISOL_LICENSE_SERVER_HOST=licenseserver
@@ -15,17 +15,15 @@ ISOL_AUTH_SERVICE_EXTERNAL_HOST=localhost
 ISOL_AUTH_SERVICE_EXTERNAL_PORT=8010
 ISOL_AUTH_SERVICE_EXTERNAL_PATH=/
 # realm in the authentication server to configure and use
-ISOL_AUTH_SERVICE_REALM=lema
+ISOL_AUTH_SERVICE_REALM=cga
 # credentials to configure and use for managing the authentication server
-ISOL_AUTH_SERVICE_ADMIN_USERNAME=lema-auth
-ISOL_AUTH_SERVICE_ADMIN_PASSWORD=lema-auth
+ISOL_AUTH_SERVICE_ADMIN_USERNAME=cga-auth
+ISOL_AUTH_SERVICE_ADMIN_PASSWORD=cga-auth
 # client and client credentials to configure and use for API requests
-ISOL_AUTH_SERVICE_API_CLIENT=lema_api
+ISOL_AUTH_SERVICE_API_CLIENT=cga
 # changes to this value will only be applied when deploying to a new realm. 
 # manually remove the existing realm from the Keycloak UI and re-run the script.
 ISOL_AUTH_SERVICE_API_CLIENT_SECRET=d0e76ad7-7d6b-4d86-be3a-5dfe715dbf87
-# client to configure and use for logging into the UI
-ISOL_AUTH_SERVICE_UI_CLIENT=lema_ui
 
 # location of the `entity` component, as accessed from Docker containers
 ISOL_ENTITY_STORAGEDB_HOST=entity-storagedb
@@ -34,37 +32,20 @@ ISOL_ENTITY_INDEXDB_HOST=entity-indexdb
 ISOL_ENTITY_INDEXDB_ACI_PORT=9100
 ISOL_ENTITY_INDEXDB_INDEX_PORT=9101
 
-# location of the `filestore` component, as accessed from Docker containers
-ISOL_FILESTORE_SERVICE_PROTOCOL=http
-ISOL_FILESTORE_SERVICE_HOST=filestore-service
-ISOL_FILESTORE_SERVICE_PORT=8000
-ISOL_FILESTORE_SERVICE_PATH=/
-# credentials to configure and use for accessing the object store
-ISOL_FILESTORE_SERVICE_USERNAME=lema-filestore
-ISOL_FILESTORE_SERVICE_PASSWORD=lema-filestore
-# bucket to use for storing files in the object store
-ISOL_FILESTORE_SERVICE_BUCKET=isol-demo-bucket
-
 # location of the `audit` component, as accessed from Docker containers
 ISOL_AUDIT_DB_HOST=audit-db
 ISOL_AUDIT_DB_PORT=5432
 # credentials to configure and use for accessing the audit database
-ISOL_AUDIT_DB_USERNAME=lema-audit
-ISOL_AUDIT_DB_PASSWORD=lema-audit
+ISOL_AUDIT_DB_USERNAME=cga-audit
+ISOL_AUDIT_DB_PASSWORD=cga-audit
 # name of the database to configure and use for storing audit logs in
-ISOL_AUDIT_DB_NAME=lema-audit
+ISOL_AUDIT_DB_NAME=cga-audit
 
 
 # changes to any of the values below will only be applied when deploying to a new realm. 
 # manually remove the existing realm from the Keycloak UI and re-run the script.
 
-# location of the `api` component, as accessed by a user
+# location of the `cga` component, as accessed by a user
 ISOL_API_SERVICE_EXTERNAL_HOST=localhost
 ISOL_API_SERVICE_EXTERNAL_PORT=8060
 ISOL_API_SERVICE_EXTERNAL_PATH=/
-
-# location of the `ui` component, as accessed by a user
-LEMA_UI_EXTERNAL_HOST=localhost
-LEMA_UI_EXTERNAL_PORT=8070
-# this must end in /
-LEMA_UI_EXTERNAL_PATH=/

config/ui.env → config/cga.env

@@ -1,2 +1,2 @@
 # address on the Docker host to expose the web server port on
-LEMA_UI_DOCKER_PORT=8070
+ISOL_API_SERVICE_DOCKER_PORT=8060

deploy.py

@@ -33,10 +33,10 @@ def wrap(text):
 
 
 DESCRIPTION = '''
-Deploy IDOL LEMA.
+Deploy IDOL CGA.
 
 ''' + wrap('''\
-This program deploys components of the LEMA system, or resumes a stopped system, or reconfigures an
+This program deploys components of the CGA system, or resumes a stopped system, or reconfigures an
 existing system.  Before running, check and update the configuration in `config/base.env`.
 ''') + '''
 
@@ -52,18 +52,14 @@ def wrap(text):
 ''') + '''
 
 - entity: storage for application data
-- analysis: media analysis system
-- analysis-live: live media analysis system
 - api: user-facing web server
-- ui: user-facing web server
 
 ''' + wrap('''\
 The components that may be deployed using this script, or may be deployed manually using suitable
 replacements, are:
 ''') + '''
 
 - auth: user-facing web server - Keycloak authentication server
-- filestore: storage for files - Amazon S3-compatible object storage
 - audit: storage for audit logs - PostgreSQL database server
 '''
 

docker-compose/docker-compose.auth-setup.yml

@@ -29,6 +29,4 @@ services:
       - ISOL_AUTH_SERVICE_API_CLIENT=${ISOL_AUTH_SERVICE_API_CLIENT}
       - ISOL_AUTH_SERVICE_API_CLIENT_SECRET=${ISOL_AUTH_SERVICE_API_CLIENT_SECRET}
       - ISOL_AUTH_SERVICE_API_URL=${ISOL_EXTERNAL_PROTOCOL}://${ISOL_API_SERVICE_EXTERNAL_HOST}:${ISOL_API_SERVICE_EXTERNAL_PORT}${ISOL_API_SERVICE_EXTERNAL_PATH}
-      - ISOL_AUTH_SERVICE_UI_CLIENT=${ISOL_AUTH_SERVICE_UI_CLIENT}
-      - ISOL_AUTH_SERVICE_UI_URL=${ISOL_EXTERNAL_PROTOCOL}://${LEMA_UI_EXTERNAL_HOST}:${LEMA_UI_EXTERNAL_PORT}${LEMA_UI_EXTERNAL_PATH}*,${ISOL_AUTH_SERVICE_ADDITIONAL_UI_URLS}
       - ISOL_AUTH_SERVICE_TOKEN_EXPIRY_TIME=${ISOL_AUTH_SERVICE_TOKEN_EXPIRY_TIME}