Skip to content

Add active column to seed CSV and update dev setup docs #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brianclemens merged 1 commit into from
Mar 2, 2026
Merged

Add active column to seed CSV and update dev setup docs #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 10 additions & 11 deletions apollo/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,25 +23,25 @@ Errata mirroring and publishing system
Setting up a development environment for Apollo involves setting up a local PostgreSQL database, Temporal server, Apollo server, and associated workers. Once you have these running, you can test changes made to the Apollo code.

## Prerequisites
1. Podman
1. PostgreSQL
1. Podman or Docker
1. PostgreSQL client (`psql`)
1. Temporal CLI: [link](https://docs.temporal.io/cli)
1. Local checkout of distro-tools
1. Bazel CLI (bazelisk): [link](https://github.com/bazelbuild/bazelisk/blob/master/README.md)

## Steps:
These steps assume you are currently in the root of the cloned distro-tools repository.
1. Create a directory for PostgreSQL DB: `mkdir -p container_data/postgres`
1. Spin up PostgreSQL container: `podman run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --security-opt=label=disable --name postgres_apollo docker.io/postgres:14`
1. Connect to PostgreSQL: `psql -U postgres -W -h 127.0.0.1`
1. Spin up PostgreSQL container:
- **Podman:** `podman run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --security-opt=label=disable --name postgres_apollo docker.io/postgres:14`
- **Docker:** `docker run -d -v $PWD/container_data/postgres:/var/lib/postgresql/data:z,rw -e POSTGRES_PASSWORD=postgres -p 5432:5432 --name postgres_apollo docker.io/postgres:14`
1. Connect to PostgreSQL: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1`
1. Once at the PostgreSQL prompt, create the database: `create database apollo2development template template0; grant all on database apollo2development to postgres;`
1. Exit PostgreSQL: `quit;`
1. Set up an alias for interfacing with the DB: `alias apollodb='psql -U postgres -W -h 127.0.0.1 apollo2development'`
1. Create needed tables: `apollodb -f apollo/schema.sql`
1. Update the last_indexed_at date to reduce the number of advisories you pull down from Red Hat: `apollodb -c "update red_hat_index_state set last_indexed_at = '2024-11-01';"` Note: you can change this date to whatever works for you.
1. Seed the needed values in the DB: `apollodb -c "\copy codes from 'apollo/db_seed/codes.csv' with DELIMITER ',';" -c "\copy supported_products from 'apollo/db_seed/supported_products.csv' with DELIMITER ',';" -c "\copy supported_products_rh_mirrors from 'apollo/db_seed/supported_products_rh_mirrors.csv' with DELIMITER ',';" -c "\copy supported_products_rpm_repomds from 'apollo/db_seed/supported_products_rpm_repomds.csv' with DELIMITER ',';"`
1. Start the Apollo server (can be run in a tmux or screen session): `bazel run apollo/server:server`
1. Create needed tables: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -f apollo/schema.sql`
1. Seed the needed values in the DB: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -c "\copy codes from 'apollo/db_seed/codes.csv' with DELIMITER ',';" -c "\copy supported_products from 'apollo/db_seed/supported_products.csv' with DELIMITER ',';" -c "\copy supported_products_rh_mirrors from 'apollo/db_seed/supported_products_rh_mirrors.csv' with DELIMITER ',';" -c "\copy supported_products_rpm_repomds from 'apollo/db_seed/supported_products_rpm_repomds.csv' with DELIMITER ',';"`
1. Start the Temporal server (can be run in a tmux or screen session): `temporal server start-dev`
1. Start the Apollo server (can be run in a tmux or screen session): `bazel run apollo/server:server`
1. Start the rhworker (can be run in a tmux or screen session): `bazel run apollo/rhworker:rhworker`
1. Start the rpmworker (can be run in a tmux or screen session): `bazel run apollo/rpmworker:rpmworker`

Expand All @@ -52,8 +52,7 @@ To start a job to ingest Red Hat advisories: `temporal workflow start --type Pol
To start a job to clone Red Hat security advisories over to Rocky: `temporal workflow start --type RhMatcherWorkflow --task-queue v2-rpmworker`

## Tips and Tricks
* Podman conflicts with Docker when it comes to the UID and GIDs that are assigned to the rootless containers, so if you've already got Docker enabled for your user, you may see issues if you attempt to run a Podman container.
* You can reduce the number of advisories pulled down from Red Hat on first sync by setting the `last_indexed_at` date: `PGPASSWORD=postgres psql -U postgres -h 127.0.0.1 apollo2development -c "update red_hat_index_state set last_indexed_at = '2024-11-01';"` — run this after seeding but before starting the workers. Adjust the date to whatever works for you.
* You can redirect the logging for a worker to a file using something like: `bazel run apollo/rpmworker:rpmworker &> rpmworker.log`
* While doing local development, you can eliminate the need to type in the PostgreSQL password by setting the `PGPASSWORD=postgres` environment variable and removing the `-W` flag from your alias or `psql` command.
* The Apollo web UI is available on port `9999`
* The Temporal web UI is available on port `8233`
40 changes: 20 additions & 20 deletions apollo/db_seed/supported_products_rh_mirrors.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
2,2023-01-31 11:10:09.117732+00,\N,1,Rocky Linux 8 x86_64,Red Hat Enterprise Linux,8,\N,x86_64
3,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 8 aarch64,Red Hat Enterprise Linux,8,\N,aarch64
4,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 x86_64,Red Hat Enterprise Linux,9,\N,x86_64
5,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 aarch64,Red Hat Enterprise Linux,9,\N,aarch64
6,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 s390x,Red Hat Enterprise Linux,9,\N,s390x
7,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 ppc64le,Red Hat Enterprise Linux,9,\N,ppc64le
8,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64
9,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64
10,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64
11,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64
12,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64
13,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64
14,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 x86_64 (Legacy),Red Hat Enterprise Linux,9,\N,x86_64
15,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 aarch64 (Legacy),Red Hat Enterprise Linux,9,\N,aarch64
16,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 s390x (Legacy),Red Hat Enterprise Linux,9,\N,s390x
17,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 ppc64le (Legacy),Red Hat Enterprise Linux,9,\N,ppc64le
18,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud x86_64,Red Hat Enterprise Linux,8,\N,x86_64
19,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud aarch64,Red Hat Enterprise Linux,8,\N,aarch64
20,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud x86_64,Red Hat Enterprise Linux,9,\N,x86_64
21,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud aarch64,Red Hat Enterprise Linux,9,\N,aarch64
2,2023-01-31 11:10:09.117732+00,\N,1,Rocky Linux 8 x86_64,Red Hat Enterprise Linux,8,\N,x86_64,true
3,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 8 aarch64,Red Hat Enterprise Linux,8,\N,aarch64,true
4,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 x86_64,Red Hat Enterprise Linux,9,\N,x86_64,true
5,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 aarch64,Red Hat Enterprise Linux,9,\N,aarch64,true
6,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 s390x,Red Hat Enterprise Linux,9,\N,s390x,true
7,2023-01-31 11:31:22.988122+00,\N,1,Rocky Linux 9 ppc64le,Red Hat Enterprise Linux,9,\N,ppc64le,true
8,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false
9,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.4 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false
10,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false
11,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.5 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false
12,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 x86_64 (Legacy),Red Hat Enterprise Linux,8,\N,x86_64,false
13,2023-01-31 12:45:02.676637+00,\N,1,Rocky Linux 8.6 aarch64 (Legacy),Red Hat Enterprise Linux,8,\N,aarch64,false
14,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 x86_64 (Legacy),Red Hat Enterprise Linux,9,\N,x86_64,false
15,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 aarch64 (Legacy),Red Hat Enterprise Linux,9,\N,aarch64,false
16,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 s390x (Legacy),Red Hat Enterprise Linux,9,\N,s390x,false
17,2023-01-31 12:46:12.173428+00,\N,1,Rocky Linux 9.0 ppc64le (Legacy),Red Hat Enterprise Linux,9,\N,ppc64le,false
18,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud x86_64,Red Hat Enterprise Linux,8,\N,x86_64,true
19,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 8 SIG Cloud aarch64,Red Hat Enterprise Linux,8,\N,aarch64,true
20,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud x86_64,Red Hat Enterprise Linux,9,\N,x86_64,true
21,2023-02-04 02:18:54.822335+00,\N,2,Rocky Linux 9 SIG Cloud aarch64,Red Hat Enterprise Linux,9,\N,aarch64,true
Loading