Skip to content

Latest commit

 

History

History
432 lines (314 loc) · 14.6 KB

File metadata and controls

432 lines (314 loc) · 14.6 KB

RELECOV Platform

RELECOV Platform deployment guide (containers and bare-metal), including iSkyLIMS and Nextstrain integration.

Get the code (required)

git clone https://github.com/bu-isciii/relecov-platform.git
cd relecov-platform

iSkyLIMS is required for the integrated platform stack:

cd ..
git clone https://github.com/bu-isciii/iskylims.git relecov-iskylims

Choose your path

  • Docker (test): full local stack for validation.
  • Docker (production): app + iSkyLIMS + Nextstrain containers pointing to your production DB.
  • Bare-metal: install directly on the host using install.sh.

Minimum requirements

Baseline software (minimum recommended):

  • MySQL 8.0+ or MariaDB 10.4+
  • Git 2.34+
  • Python 3.11+
  • Apache HTTP Server 2.4+ (for reverse proxy on production)
  • Docker Engine + Compose v2, or Podman 4.9+ + podman-compose
  • tar, rsync, wget, curl

Notes:

  • This guide does not cover MySQL installation itself, only DB/user/grant creation and operational tasks.
  • For container deployments with external DB, ensure network connectivity from containers to DB host/port.
  • For Podman, prefer host.containers.internal in DB host settings.

Docker deployment

Prerequisites:

  • Docker Compose v2 or Podman + podman-compose.
  • Both repositories checked out side-by-side:
    • relecov-platform
    • relecov-iskylims

Local test stack

Runs:

  • relecov_test_db (MySQL)
  • relecov_test_app (RELECOV Platform)
  • relecov_test_iskylims_app (iSkyLIMS)
  • relecov_test_nextstrain (Nextstrain viewer)

Basic test install:

cd relecov-platform
bash container_install.sh --test --engine podman 2>&1 | tee relecov_test_install.log

Per-service conf mapping (recommended):

bash container_install.sh --test --engine podman \
  --install_conf_map app,conf/docker_test_platform_settings.txt \
  --install_conf_map iskylims_app,conf/docker_test_iskylims_settings.txt \
  2>&1 | tee relecov_test_install.log

The container images now include the staged application trees for both relecov-platform and relecov-iskylims. Test containers can therefore be recreated or restarted without rerunning the file installation step; container_install.sh only runs the runtime bootstrap tasks inside each container.

Production container stack

  1. Prepare production confs (separate for each service):
cp conf/docker_production_settings.txt conf/docker_production_platform_settings.txt
cp ../relecov-iskylims/conf/docker_production_settings.txt ../relecov-iskylims/conf/docker_production_iskylims_settings.txt
  1. Edit both files with your production DB/network values.

  2. Deploy:

cd relecov-platform
bash container_install.sh --engine podman \
  --action install \
  --install_conf_map app,conf/docker_production_platform_settings.txt \
  --install_conf_map iskylims_app,../relecov-iskylims/conf/docker_production_iskylims_settings.txt \
  2>&1 | tee relecov_prod_install.log
  1. Endpoints:
  • RELECOV Platform: http://<host>:8000
  • iSkyLIMS: http://<host>:8001
  • Nextstrain: http://<host>:8100

Production images now bake the staged Django application trees into the images themselves. Host reboots or container recreation no longer require rerunning the app installation step; container_install.sh only performs runtime bootstrap tasks such as migrations, optional scripts, superuser creation on first install, and collectstatic.

Persist logs/documents on the host

For production container deployments, keep these paths persistent:

  • Logs: /var/log/local/apps/relecov-platform -> /opt/relecov-platform/logs
  • Apache logs: /var/log/local/apache -> /var/log/httpd
  • Apache config: ${APP_INSTALL_PATH:-/opt/relecov-platform}/conf/relecov_apache_reverse_proxy.conf -> /etc/httpd/conf.d/relecov.conf
  • Documents: named volume relecov_documents -> /opt/relecov-platform/documents
  • Static: /opt/relecov-platform/static-host -> /opt/relecov-platform/static

Prepare host directories and ownership:

sudo mkdir -p /var/log/local/apps/relecov-platform
sudo mkdir -p /var/log/local/apache
sudo mkdir -p ${APP_INSTALL_PATH:-/opt/relecov-platform}/conf
sudo mkdir -p /opt/relecov-platform/static-host
sudo chown -R ${APP_UID:-1212}:${APP_GID:-1212} /var/log/local/apps/relecov-platform /opt/relecov-platform/static-host ${APP_INSTALL_PATH:-/opt/relecov-platform}

Apache reverse proxy (container) + Gunicorn

For production, the app container runs gunicorn and the apache service in docker-compose.prod.yml acts as the reverse proxy for RELECOV Platform, iSkyLIMS, and Nextstrain.

During container_install.sh, the file conf/relecov_apache_reverse_proxy.conf is copied to ${APP_INSTALL_PATH:-/opt/relecov-platform}/conf/relecov_apache_reverse_proxy.conf on the host before compose up. Runtime Apache logs are written to /var/log/local/apache.

If you need a different runtime root, export APP_INSTALL_PATH before running container_install.sh. The compose file already mounts the Apache config from ${APP_INSTALL_PATH}.

Upgrade docker deployment

Use upgrade when DB schema/data already exist.

IMPORTANT:Before upgrading, perform backups from Backups.

cd relecov-platform
bash container_install.sh --engine podman \
  --action upgrade \
  --install_conf_map app,conf/docker_production_platform_settings.txt \
  --install_conf_map iskylims_app,../relecov-iskylims/conf/docker_production_iskylims_settings.txt \
  2>&1 | tee relecov_prod_upgrade.log

The upgrade path rebuilds/restarts the containers and runs install.sh --bootstrap upgrade inside each app container. The app files are already baked into the rebuilt images; the bootstrap phase applies migrations with --fake-initial, refreshes static files, and skips first-install setup.

Bare-metal deployment (Ubuntu/CentOS)

Install

  1. Prepare DB/users/grants from Database creation, users and grants.
  2. Edit config files:
  • relecov-platform/conf/install_settings.txt
  • relecov-iskylims/conf/install_settings.txt
  1. Install dependencies + app (needs root):
cd relecov-platform
bash install.sh --install full --conf conf/install_settings.txt

cd ../relecov-iskylims
bash install.sh --install full --conf conf/install_settings.txt

The staged/bootstrap split added for container images is internal. Bare-metal commands do not change: --install and --upgrade still run the complete dependency, application, and database workflow shown here.

Upgrade

cd relecov-platform
bash install.sh --upgrade app --conf conf/install_settings.txt

cd ../relecov-iskylims
bash install.sh --upgrade app --conf conf/install_settings.txt

Before upgrading, perform backups from Backups.

Common operations (Docker + bare-metal)

Database creation, users and grants

Run as MySQL root:

CREATE DATABASE IF NOT EXISTS relecov_dev CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE DATABASE IF NOT EXISTS iskylims_rel_dev CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

CREATE USER IF NOT EXISTS 'relecov'@'%' IDENTIFIED BY 'djangopass';
CREATE USER IF NOT EXISTS 'relecovlims'@'%' IDENTIFIED BY 'djangopass';

CREATE USER IF NOT EXISTS 'relecov'@'localhost' IDENTIFIED BY 'djangopass';
CREATE USER IF NOT EXISTS 'relecovlims'@'localhost' IDENTIFIED BY 'djangopass';

GRANT ALL PRIVILEGES ON relecov_dev.* TO 'relecov'@'%';
GRANT ALL PRIVILEGES ON relecov_dev.* TO 'relecov'@'localhost';

GRANT ALL PRIVILEGES ON iskylims_rel_dev.* TO 'relecovlims'@'%';
GRANT ALL PRIVILEGES ON iskylims_rel_dev.* TO 'relecovlims'@'localhost';

FLUSH PRIVILEGES;

Verification:

SHOW GRANTS FOR 'relecov'@'%';
SHOW GRANTS FOR 'relecovlims'@'%';

Backups

Database dumps:

mysqldump -h <db_host> -P <db_port> -u relecov -p relecov_dev > relecov_dev_$(date +%Y%m%d_%H%M%S).sql
mysqldump -h <db_host> -P <db_port> -u relecovlims -p iskylims_rel_dev > iskylims_rel_dev_$(date +%Y%m%d_%H%M%S).sql

RELECOV logs (host bind path used by production compose):

tar -czf relecov_logs_$(date +%Y%m%d_%H%M%S).tgz -C /var/log/local/apps/relecov-platform .

Docker/Podman named volumes (documents + nextstrain datasets):

# podman (replace with docker if needed)
podman run --rm -v relecov_documents:/from -v "$PWD":/to alpine \
  tar -czf /to/relecov_documents_$(date +%Y%m%d_%H%M%S).tgz -C /from .

podman run --rm -v nextstrain_data:/from -v "$PWD":/to alpine \
  tar -czf /to/nextstrain_data_$(date +%Y%m%d_%H%M%S).tgz -C /from .

Suggested backup order before any risky change:

  1. DB dumps.
  2. Documents volume.
  3. Nextstrain datasets volume.
  4. Logs archive.

Restore / rollback

Restore DB:

mysql -h <db_host> -P <db_port> -u relecov -p relecov_dev < relecov_dev_YYYYMMDD_HHMMSS.sql
mysql -h <db_host> -P <db_port> -u relecovlims -p iskylims_rel_dev < iskylims_rel_dev_YYYYMMDD_HHMMSS.sql

Restore volumes:

podman run --rm -v relecov_documents:/to -v "$PWD":/from alpine \
  sh -lc "cd /to && tar -xzf /from/relecov_documents_YYYYMMDD_HHMMSS.tgz"

podman run --rm -v nextstrain_data:/to -v "$PWD":/from alpine \
  sh -lc "cd /to && tar -xzf /from/nextstrain_data_YYYYMMDD_HHMMSS.tgz"

Restore logs archive:

sudo mkdir -p /var/log/local/apps/relecov-platform
sudo tar -xzf relecov_logs_YYYYMMDD_HHMMSS.tgz -C /var/log/local/apps/relecov-platform

What to do if something fails

Quick checks:

podman ps -a
podman logs --tail 200 relecov_app
podman logs --tail 200 relecov_iskylims_app
podman logs --tail 200 relecov_nextstrain

If deployment is broken after migration or bad data load, rollback using Restore / rollback.

Post-install configuration

RELECOV Platform

  1. Create the admin/superuser if not already created.
  2. Load RELECOV schema: in Configuration -> SchemaHandling, upload the latest relecov_schema.json: https://raw.githubusercontent.com/BU-ISCIII/relecov-tools/main/relecov_tools/schema/relecov_schema.json
  3. Load annotation GFF: in Configuration -> Annotation, upload: conf/NC_045512.2.gff
  4. iSkyLIMS endpoint verification: in /admin/core/configsetting/, ensure ISKYLIMS_SERVER points to your iSkyLIMS service (for example relecov-lims in test, or your production hostname).

iSkyLIMS

Pre-load RELECOV sample project data before normal usage:

cd /opt/iskylims
source virtualenv/bin/activate
python manage.py migrate --noinput
python manage.py loaddata /path/to/relecov_iskylims_projects.json

If you need to generate that fixture from an existing environment:

cd /opt/iskylims
source virtualenv/bin/activate
python manage.py dumpdata --indent 2 core.SampleProjects > ~/relecov_iskylims_test_data.json

Then configure RELECOV in iSkyLIMS:

  1. Create relecovbot in /admin/auth/user/ and assign required permissions/groups for your integration flows.
  2. Go to WetLab (/wetlab) and open: PARAMETERS SETTINGS -> Define Sample projects.
  3. If the project does not exist, create one including RELECOV in the name and set the manager (for example isabel.cuesta in your environment).
  4. In that RELECOV project: Define fields -> Load batch, then set:
  • upload schema: relecov_schema.json
  • Property where to fetch fields: classification
  • select classifications to import:
    • Database Identifiers
    • Files info
    • Host information
    • Pathogen diagnostic testing
    • Sample collection and processing
    • Sequencing
  1. Configure ontology mapping in /admin/core/ontologymap/:
  • lab_request -> GENEPIO:0001153 (submitting_institution)
  • species -> GENEPIO:0001386 (Organism)
  • sample_name -> GENEPIO:0001123 (sequencing_sample_id)
  • sample_entry_date -> NCIT:C93644 (received_date)
  • collection_sample_date -> GENEPIO:0001174 (sample_collection_date)
  1. Define species in: PARAMETERS SETTINGS -> Initial Settings -> Define new Specie. Add all enum values used by your schema for Organism (or the ontology-mapped equivalent), for example:
  • Severe acute respiratory syndrome coronavirus 2
  • Respiratory syncytial virus
  • Influenza virus
  1. Ensure wetlab app assignment is enabled for the related flow.

Smoke test

Recommended endpoint checks:

curl -I http://<host>:8000
curl -I http://<host>:8001
curl -I http://<host>:8100

Developer workflow

Migrations

Create and review migrations locally, then commit migration files:

cd /opt/relecov-platform
source virtualenv/bin/activate

python manage.py makemigrations core dashboard
python manage.py migrate --noinput
python manage.py showmigrations core dashboard

Do not rely on runtime auto-generation of migrations in production.

Container-based development loop

# bring stack down (keep volumes)
podman-compose -f docker-compose.test.yml down

# rebuild changed services
podman-compose -f docker-compose.test.yml build app iskylims_app

# start again
podman-compose -f docker-compose.test.yml up -d

Persistent host paths

See Persist logs/documents on the host in the production deployment section.

Useful diagnostics

podman exec -it relecov_test_app python /opt/relecov-platform/manage.py check
podman exec -it relecov_test_iskylims_app python /opt/iskylims/manage.py check
podman volume ls | grep -E 'relecov|nextstrain'