Fix per node example script

[zguig52]

Fix issue #6

[mmols]

Hi @zguig52 - these examples are built around leveraging the hostnames provided to services in docker-compose - so postgres-n1 / postgres-n2 are the correct hostnames that we want to wire up subscriptions for in spock.

The purpose of these examples are to quickly try out these images (postgres + extensions) without really changing anything - the distributed one is primarily for giving you a quick path to trying out spock.

The issue referenced in this PR seems to be providing hostnames / specific IPs to some of these variables, which seems a bit beyond the intent of the examples. These aren't really for production usage.

If you can tell me a bit more about how you are using or have adapted this example, I can consider incorporating it into the examples without breaking their purpose, or suggest another tool for you that might be easier to use.

One idea to perhaps support solving this issue would be to support a NODE_HOST env variable on each node, and use that in this line (with a default), rather than the node names.

[zguig52]

Hi @mmols ,

Thanks for your feedback. I just finished testing deployment of pgEdge, which seems a promising "Galera like" extension for PostgreSQL. Thanks for your work on this project. I will test / use it more closely in coming days.

I started by following a bit "blindly" the examples from the repo but had to rework it a bit. My target was to deploy a single PG instance for multiple DB / projects.

Hereafter are some feedbacks and improvements I did that could simplify pgEdge deployment / use in container contexts.

1. From multiple scripts to a single script + customized configuration files

I changed the scripts to customized postgresql.conf + pg_hba.conf, including all customized configuration for Spock.

For pg_hba, I changed the deprecated md5 authentication scheme to the current scram-sha-256 scheme for (could be customized for less open private networks):

local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            scram-sha-256
host    all             all             172.16.0.0/12            scram-sha-256
host    all             all             10.0.0.0/8            scram-sha-256

For postgresql.conf, I directly included extensions required load + spock requirements, which are these non default parameters:

listen_addresses = '*'         # what IP address(es) to listen on;
port = {{ databases.postgresql.port }}                            # (change requires restart)
max_worker_processes = 10               # (change requires restart)
wal_level = logical                    # minimal, replica, or logical
track_commit_timestamp = on   # collect timestamp of transaction commit
shared_preload_libraries = 'pg_stat_statements,pgaudit,snowflake,spock,postgis-3'          # (change requires restart)
#------------------------------------------------------------------------------
# SPOCK
#------------------------------------------------------------------------------
spock.enable_ddl_replication = on
spock.include_ddl_repset = on
spock.allow_ddl_from_functions = on
spock.conflict_resolution = 'last_update_wins'
spock.save_resolutions = on
spock.conflict_log_level = 'DEBUG'

For the initialisation, I adapted the example scripts like this, by using the same current node syntax as for all nodes syntax:

#!/usr/bin/env bash
# Inspired by scripts here: https://github.com/pgEdge/postgres-images/blob/main/examples/compose/distributed/docker-compose.yaml
set -Eeo pipefail

### VARS INIT

: "${NODE:?NODE required e.g NODE='n1:postgres-n1:5432'}"
: "${NODES:?NODES list is required, e.g. NODES='n1:postgres-n1:5432,n2:postgres-n2:5432'}"
: "${POSTGRES_PASSWORD:?POSTGRES_PASSWORD required}"
: "${PGEDGE_USER:?PGEDGE_USER required}"
: "${PGEDGE_PASSWORD:?PGEDGE_PASSWORD required}"

# Get overloaded or set default values
POSTGRES_USER="${POSTGRES_USER:-postgres}"
POSTGRES_DB="${POSTGRES_DB:-postgres}"

# Admin DB
ADMIN_DB="${POSTGRES_DB}"

# Target DB
DB="${NEW_DB:-$POSTGRES_DB}"

# Admin credentials (local)
ADMIN_USER="${POSTGRES_USER}"
ADMIN_PASSWORD="${POSTGRES_PASSWORD}"
export PGPASSWORD="${ADMIN_PASSWORD}"

# Target user (local)
USER="${NEW_USER:-$POSTGRES_USER}"
USER_PASSWORD="${NEW_PASSWORD:-$POSTGRES_PASSWORD}"

# Replication user (used in DSNs)
PGEDGE_USER="${PGEDGE_USER}"
PGEDGE_PASSWORD="${PGEDGE_PASSWORD}"

# Parse current node configuration
NODE_NAME="${NODE%%:*}"
rest="${NODE#*:}"
NODE_HOST="${rest%%:*}"
NODE_PORT="${rest##*:}"

# Parse spock cluster nodes configuration
IFS=',' read -r -a NODE_ENTRIES <<< "$NODES"
NAMES=(); HOSTS=(); PORTS=()
for entry in "${NODE_ENTRIES[@]}"; do
    name="${entry%%:*}"
    rest="${entry#*:}"
    host="${rest%%:*}"
    port="${rest##*:}"
    NAMES+=("$name"); HOSTS+=("$host"); PORTS+=("$port")
done

# Check if DB is listening or is in bootstrap / local socket only
PG_LISTENING=$(psql -v ON_ERROR_STOP=1 -U "$ADMIN_USER" -p "$NODE_PORT" -d "$ADMIN_DB" -tAc "SELECT 1;" || true)
PSQL_CMD=(psql -v ON_ERROR_STOP=1 -U "$ADMIN_USER")
if [ "$PG_LISTENING" == "1" ]; then
    PSQL_CMD+=( -p "$NODE_PORT")
fi

### 0 ### CREATE DB AND USER IF NOT EXISTING

echo "Ensuring '$USER' exists and has right password configured..."
USER_EXISTS=$("${PSQL_CMD[@]}" -d "$ADMIN_DB" -tAc "SELECT 1 FROM pg_roles WHERE rolname='$USER';" || true)
if [ "$USER_EXISTS" != "1" ]; then
    "${PSQL_CMD[@]}" -d "$ADMIN_DB" \
        -c "CREATE ROLE \"$USER\" LOGIN PASSWORD '$USER_PASSWORD';"
else
    if [ "$USER" == "ADMIN_USER" ]; then
        "${PSQL_CMD[@]}" -d "$ADMIN_DB" \
            -c "ALTER ROLE \"$USER\" SUPERUSER PASSWORD '$USER_PASSWORD';"
    else
        "${PSQL_CMD[@]}" -d "$ADMIN_DB" \
            -c "ALTER ROLE \"$USER\" LOGIN PASSWORD '$USER_PASSWORD';"
    fi
fi

echo "Ensuring database '$DB' exists with owner '$USER'..."
DB_EXISTS=$("${PSQL_CMD[@]}" -d "$ADMIN_DB" -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB';" || true)
if [ "$DB_EXISTS" != "1" ]; then
    "${PSQL_CMD[@]}" -d "$ADMIN_DB" \
        -c "CREATE DATABASE \"$DB\" OWNER \"$USER\";"
else
    "${PSQL_CMD[@]}" -d "$DB" \
        -c "ALTER DATABASE \"$DB\" OWNER TO \"$USER\";"
fi

### 1 ### CREATE EXTENSIONS

EXTENSIONS=("pg_stat_statements" "pgaudit" "snowflake" "spock" "vector" "postgis")

echo "Initializing extensions: ${EXTENSIONS[*]}"
for EXT in "${EXTENSIONS[@]}"; do
    echo "Creating extension: $EXT"
    "${PSQL_CMD[@]}" -d "$DB" -c "CREATE EXTENSION IF NOT EXISTS \"$EXT\";"
done

### 2 ### SETUP SPOCK
# Per-node setup: also ensures the pgedge role exists; node_create uses pgedge in DSN.

echo "[spock-node] Ensuring replication role '$PGEDGE_USER' exists and is configured..."
EXISTS=$("${PSQL_CMD[@]}" -d "$DB" -tAc "SELECT 1 FROM pg_roles WHERE rolname = '$PGEDGE_USER' LIMIT 1;" || true)
if [ "$EXISTS" != "1" ]; then
    "${PSQL_CMD[@]}" -d "$DB" \
        -c "CREATE ROLE \"$PGEDGE_USER\" LOGIN REPLICATION PASSWORD '$PGEDGE_PASSWORD';"
else
    "${PSQL_CMD[@]}" -d "$DB" \
        -c "ALTER ROLE \"$PGEDGE_USER\" LOGIN REPLICATION PASSWORD '$PGEDGE_PASSWORD';"
fi

# Practical privileges (safe to re-run)
"${PSQL_CMD[@]}" -d "$DB" -c "GRANT pg_read_all_data  TO \"$PGEDGE_USER\";"
"${PSQL_CMD[@]}" -d "$DB" -c "GRANT pg_write_all_data TO \"$PGEDGE_USER\";"
"${PSQL_CMD[@]}" -d "$DB" -c "GRANT CREATE, TEMP ON DATABASE \"$DB\" TO \"$PGEDGE_USER\";"
"${PSQL_CMD[@]}" -d "$DB" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES   TO \"$PGEDGE_USER\";"
"${PSQL_CMD[@]}" -d "$DB" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO \"$PGEDGE_USER\";"
"${PSQL_CMD[@]}" -d "$DB" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON FUNCTIONS TO \"$PGEDGE_USER\";"

echo "[spock-node] Ensuring spock node '${NODE_NAME}' exists (dsn uses '${PGEDGE_USER}')..."
NODE_EXISTS=$("${PSQL_CMD[@]}" -d "$DB" -tAc "SELECT 1 FROM spock.node WHERE node_name='${NODE_NAME}' LIMIT 1;" || true)
if [ "$NODE_EXISTS" != "1" ]; then
    "${PSQL_CMD[@]}" -d "$DB" \
        -c "SELECT spock.node_create(node_name := '${NODE_NAME}', dsn := 'host=${NODE_HOST} port=${NODE_PORT} dbname=${DB} user=${PGEDGE_USER} password=${PGEDGE_PASSWORD}');"
    echo "[spock-node] node '${NODE_NAME}' created."
else
    echo "[spock-node] node '${NODE_NAME}' already present; skipping."
fi

### 3 ### WIRING SCRIPT
# Wiring script: admin executes sub_create; provider_dsn uses replication user.

psql_ok () {
    local host="$1" port="$2" user="$3"
    psql -h "$host" -p "$port" -U "$user" -d "$DB" -t -A -v ON_ERROR_STOP=1 -c "select 1" >/dev/null 2>&1
}

if [ "$PG_LISTENING" == "1" ]; then

    echo "[wire] Waiting for all nodes (admin access)..."
    for i in "${!NAMES[@]}"; do
        h="${HOSTS[$i]}"; p="${PORTS[$i]}"
        until psql_ok "$h" "$p" "$ADMIN_USER"; do
            echo "[wire] waiting for $h:$p ..."
            sleep 1
        done
    done

    echo "[wire] Waiting until each DB reports its own spock.node row..."
    for i in "${!NAMES[@]}"; do
        n="${NAMES[$i]}"; h="${HOSTS[$i]}"; p="${PORTS[$i]}"
        until psql -h "$h" -p "$p" -U "$ADMIN_USER" -d "$DB" -t -A \
            -c "SELECT 1 FROM spock.node WHERE node_name='$n' LIMIT 1;" | grep -q '^1$'; do
            echo "[wire] waiting for spock.node '$n' on $h:$p ..."
            sleep 1
        done
    done

    # Wait if not first node of the list (to let time for subscription to be replicated on other nodes and avoid error)
    if [ "${NAMES[0]}" !=  "$NODE_NAME" ]; then
        sleep 5
    fi

    echo "[wire] Creating full-mesh subscriptions (idempotent; providers use '$PGEDGE_USER')..."
    for i in "${!NAMES[@]}"; do
        ai="${NAMES[$i]}"; ah="${HOSTS[$i]}"; ap="${PORTS[$i]}"
        for j in "${!NAMES[@]}"; do
            [ "$i" -eq "$j" ] && continue
            bj="${NAMES[$j]}"; bh="${HOSTS[$j]}"; bp="${PORTS[$j]}"
            sub="sub_${ai}_${bj}"
            echo "[wire] ensuring $sub exists on $ai -> $bj ..."
            psql -h "$ah" -p "$ap" -U "$ADMIN_USER" -d "$DB" -v ON_ERROR_STOP=1 \
                -c "SELECT spock.sub_create(subscription_name := '$sub', provider_dsn := 'host=$bh port=$bp dbname=$DB user=$PGEDGE_USER password=$PGEDGE_PASSWORD') WHERE NOT EXISTS (SELECT 1 FROM spock.subscription WHERE sub_name='$sub');" \
                >/dev/null
        done
    done
    echo "[wire] Wiring complete."
else
    echo "[wire] Skip DB wiring (PG_LISTENING: $PG_LISTENING)..."
fi

With these 3 files, deployment is quite easy, with following configuration (example for podman rootless quadlet files).

databases.network:

[Network]
Subnet={{ network | ansible.utils.ipsubnet( config.network.subnet_netmask_length, 210) }}
Gateway={{ network | ansible.utils.ipsubnet( config.network.subnet_netmask_length, 210) | ansible.utils.ipmath(1) }}

postgresql.pod:

[Unit]
Description=PostgreSQL Pod
Wants=network-online.target wg-quick@{{ vpn.interface }}.service
After=network-online.target wg-quick@{{ vpn.interface }}.service

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

[Pod]
Network=databases.network:ip={{ network | ansible.utils.ipsubnet( config.network.subnet_netmask_length, 210) | ansible.utils.ipmath(11) }}
PodmanArgs=--cpus=2
PodmanArgs=--hostname=postgresql.{{ hostname }}
PodName=postgresql
PodmanArgs=--hosts-file=none
UserNS=keep-id:uid=26,gid=26
PublishPort={{ vpn_gateway }}:{{ databases.postgresql.port }}:{{ databases.postgresql.port }}
{% for peer in groups.all %}
AddHost=postgresql.{{ hostvars[peer].hostname }}:{{ hostvars[peer].vpn_gateway }}
{% endfor %}

# TEMP FIX FOR BUG HERE: https://github.com/containers/podman/issues/25596
PodmanArgs=--exit-policy=continue

postgresql.container:

[Unit]
Description=PostgreSQL container

[Service]
Restart=always
RestartSec=10
TimeoutStopSec=180

[Container]
Label=app=postgresql
ContainerName=postgresql
Image=ghcr.io/pgedge/pgedge-postgres:{{ databases.postgresql.pgedge_release }}
Pod=postgresql.pod
ReadOnly=true
Timezone=local
Volume={{ databases.folder }}/postgresql/config/scripts/init.sh:/docker-entrypoint-initdb.d/init.sh:ro
Volume={{ databases.folder }}/postgresql/config/scripts:/scripts:ro
Volume={{ databases.folder }}/postgresql/data:/var/lib/pgsql:z
Volume={{ databases.folder }}/postgresql/config/pg_hba.conf:/etc/postgresql/pg_hba.conf:ro
Volume={{ databases.folder }}/postgresql/config/postgresql.conf:/etc/postgresql/postgresql.conf:ro
Environment=POSTGRES_USER={{ databases.postgresql.users.admin.username }}
Environment=POSTGRES_PASSWORD={{ databases.postgresql.users.admin.password }}
Environment=POSTGRES_DB=postgres
Environment=PGEDGE_USER={{ databases.postgresql.users.replication.username }}
Environment=PGEDGE_PASSWORD={{ databases.postgresql.users.replication.password }}
Environment=NODE={{ hostname | regex_replace("[^a-z0-9]", "_") }}:postgresql.{{ hostname }}:{{ databases.postgresql.port }}
Environment=NODES={% for peer in groups.all %}{{ hostvars[peer].hostname | regex_replace("[^a-z0-9]", "_") }}:postgresql.{{ hostvars[peer].hostname }}:{{ databases.postgresql.port }}{% if not loop.last %},{% endif %}{% endfor %} 
Exec=-c config_file=/etc/postgresql/postgresql.conf -c hba_file=/etc/postgresql/pg_hba.conf

HealthCmd=pg_isready -U {{ databases.postgresql.users.admin.username }} -d pgedgedb -h 127.0.0.1 -p {{ databases.postgresql.port }}
HealthRetries=12
HealthInterval=5s
HealthTimeout=5s
HealthOnFailure=kill
HealthStartPeriod=60s

2. Initialization + reuse

Once the containers are mounted with the custom conf and init script inside auto bootstrap folder, once all nodes are started, I is only needed to call again the same to finish the wiring.

Another advantage is that you can also call the same script with 3 other env vars, to create a new user/password + db for any project:

- name: Test create new db and user
  become: true
  become_user: "{{ databases.user.name }}"
  containers.podman.podman_container_exec:
    name: postgresql
    command: /scripts/init.sh
    env:
      NEW_DB: testdb
      NEW_USER: testuser
      NEW_PASSWORD: testpwd

What do you think of this strategy?

3. Discussion on using template1 DB.

Have you already tried to pass extensions + spock related SQL to the template1 DB, so when creating a new DB, everything is already prepared for multi-master use?

4. Would you be open to a contribution where I adapt this stragegy for docker-compose (or including some modifications / advices you might have) to make the bootstrap much easier + some README / doc improvments I thought?