diff --git a/apollo/README.md b/apollo/README.md index 7376887..33eaad4 100644 --- a/apollo/README.md +++ b/apollo/README.md @@ -23,8 +23,8 @@ Errata mirroring and publishing system Setting up a development environment for Apollo involves setting up a local PostgreSQL database, Temporal server, Apollo server, and associated workers. Once you have these running, you can test changes made to the Apollo code. ## Prerequisites -1. Podman -1. PostgreSQL +1. Podman or Docker +1. PostgreSQL client (`psql`) 1. Temporal CLI: [link](https://docs.temporal.io/cli) 1. Local checkout of distro-tools 1. Bazel CLI (bazelisk): [link](https://github.com/bazelbuild/bazelisk/blob/master/README.md) @@ -32,16 +32,16 @@ Setting up a development environment for Apollo involves setting up a local Post ## Steps: These steps assume you are currently in the root of the cloned distro-tools repository. 1. Create a directory for PostgreSQL DB: `mkdir -p container_data/postgres` -1. Spin up PostgreSQL container: `podman run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --security-opt=label=disable --name postgres_apollo docker.io/postgres:14` -1. Connect to PostgreSQL: `psql -U postgres -W -h 127.0.0.1` +1. Spin up PostgreSQL container: + - **Podman:** `podman run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --security-opt=label=disable --name postgres_apollo docker.io/postgres:14` + - **Docker:** `docker run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --name postgres_apollo docker.io/postgres:14` +1. Connect to PostgreSQL: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1` 1. Once at the PostgreSQL prompt, create the database: `create database apollo2development template template0; grant all on database apollo2development to postgres;` 1. Exit PostgreSQL: `quit;` -1. Set up an alias for interfacing with the DB: `alias apollodb='psql -U postgres -W -h 127.0.0.1 apollo2development'` -1. Create needed tables: `apollodb -f apollo/schema.sql` -1. Update the last_indexed_at date to reduce the number of advisories you pull down from Red Hat: `apollodb -c "update red_hat_index_state set last_indexed_at = '2024-11-01';"` Note: you can change this date to whatever works for you. -1. Seed the needed values in the DB: `apollodb -c "\copy codes from 'apollo/db_seed/codes.csv' with DELIMITER ',';" -c "\copy supported_products from 'apollo/db_seed/supported_products.csv' with DELIMITER ',';" -c "\copy supported_products_rh_mirrors from 'apollo/db_seed/supported_products_rh_mirrors.csv' with DELIMITER ',';" -c "\copy supported_products_rpm_repomds from 'apollo/db_seed/supported_products_rpm_repomds.csv' with DELIMITER ',';"` -1. Start the Apollo server (can be run in a tmux or screen session): `bazel run apollo/server:server` +1. Create needed tables: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -f apollo/schema.sql` +1. Seed the needed values in the DB: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -c "\copy codes from 'apollo/db_seed/codes.csv' with DELIMITER ',';" -c "\copy supported_products from 'apollo/db_seed/supported_products.csv' with DELIMITER ',';" -c "\copy supported_products_rh_mirrors from 'apollo/db_seed/supported_products_rh_mirrors.csv' with DELIMITER ',';" -c "\copy supported_products_rpm_repomds from 'apollo/db_seed/supported_products_rpm_repomds.csv' with DELIMITER ',';"` 1. Start the Temporal server (can be run in a tmux or screen session): `temporal server start-dev` +1. Start the Apollo server (can be run in a tmux or screen session): `bazel run apollo/server:server` 1. Start the rhworker (can be run in a tmux or screen session): `bazel run apollo/rhworker:rhworker` 1. Start the rpmworker (can be run in a tmux or screen session): `bazel run apollo/rpmworker:rpmworker` @@ -52,8 +52,7 @@ To start a job to ingest Red Hat advisories: `temporal workflow start --type Pol To start a job to clone Red Hat security advisories over to Rocky: `temporal workflow start --type RhMatcherWorkflow --task-queue v2-rpmworker` ## Tips and Tricks -* Podman conflicts with Docker when it comes to the UID and GIDs that are assigned to the rootless containers, so if you've already got Docker enabled for your user, you may see issues if you attempt to run a Podman container. +* You can reduce the number of advisories pulled down from Red Hat on first sync by setting the `last_indexed_at` date: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -c "update red_hat_index_state set last_indexed_at = '2024-11-01';"` — run this after seeding but before starting the workers. Adjust the date to whatever works for you. * You can redirect the logging for a worker to a file using something like: `bazel run apollo/rpmworker:rpmworker &> rpmworker.log` -* While doing local development, you can eliminate the need to type in the PostgreSQL password by setting the `PGPASSWORD=postgres` environment variable and removing the `-W` flag from your alias or `psql` command. * The Apollo web UI is available on port `9999` * The Temporal web UI is available on port `8233` diff --git a/apollo/db_seed/supported_products_rh_mirrors.csv b/apollo/db_seed/supported_products_rh_mirrors.csv index 2cc4507..aea4cc7 100644 --- a/apollo/db_seed/supported_products_rh_mirrors.csv +++ b/apollo/db_seed/supported_products_rh_mirrors.csv @@ -1,20 +1,20 @@ -2,2023-01-31 11:10:09.117732+00,\N,1,Rocky Linux 8 x86_64,Red Hat Enterprise Linux,8,\N,x86_64 -3,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 8 aarch64,Red Hat Enterprise Linux,8,\N,aarch64 -4,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 x86_64,Red Hat Enterprise Linux,9,\N,x86_64 -5,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 aarch64,Red Hat Enterprise Linux,9,\N,aarch64 -6,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 s390x,Red Hat Enterprise Linux,9,\N,s390x -7,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 ppc64le,Red Hat Enterprise Linux,9,\N,ppc64le -8,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64 -9,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64 -10,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64 -11,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64 -12,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64 -13,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64 -14,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 x86_64 (Legacy),Red Hat Enterprise Linux,9,\N,x86_64 -15,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 aarch64 (Legacy),Red Hat Enterprise Linux,9,\N,aarch64 -16,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 s390x (Legacy),Red Hat Enterprise Linux,9,\N,s390x -17,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 ppc64le (Legacy),Red Hat Enterprise Linux,9,\N,ppc64le -18,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud x86_64,Red Hat Enterprise Linux,8,\N,x86_64 -19,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud aarch64,Red Hat Enterprise Linux,8,\N,aarch64 -20,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud x86_64,Red Hat Enterprise Linux,9,\N,x86_64 -21,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud aarch64,Red Hat Enterprise Linux,9,\N,aarch64 +2,2023-01-31 11:10:09.117732+00,\N,1,Rocky Linux 8 x86_64,Red Hat Enterprise Linux,8,\N,x86_64,true +3,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 8 aarch64,Red Hat Enterprise Linux,8,\N,aarch64,true +4,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 x86_64,Red Hat Enterprise Linux,9,\N,x86_64,true +5,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 aarch64,Red Hat Enterprise Linux,9,\N,aarch64,true +6,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 s390x,Red Hat Enterprise Linux,9,\N,s390x,true +7,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 ppc64le,Red Hat Enterprise Linux,9,\N,ppc64le,true +8,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false +9,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false +10,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false +11,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false +12,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false +13,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false +14,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 x86_64 (Legacy),Red Hat Enterprise Linux,9,\N,x86_64,false +15,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 aarch64 (Legacy),Red Hat Enterprise Linux,9,\N,aarch64,false +16,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 s390x (Legacy),Red Hat Enterprise Linux,9,\N,s390x,false +17,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 ppc64le (Legacy),Red Hat Enterprise Linux,9,\N,ppc64le,false +18,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud x86_64,Red Hat Enterprise Linux,8,\N,x86_64,true +19,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud aarch64,Red Hat Enterprise Linux,8,\N,aarch64,true +20,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud x86_64,Red Hat Enterprise Linux,9,\N,x86_64,true +21,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud aarch64,Red Hat Enterprise Linux,9,\N,aarch64,true