gerrit container not starting

[s-friedman]

I've verified that gerrit-docker is up to date and has had no modifications from the stock repo. Trying to start the gerrit container yields:

FATA[0001] Error response from daemon: Cannot start container 936f9931e733f30457269055785653bda80b4dd1d9c650d771d117c688e11a2f: Cannot link to a non running container: /gerrit AS /proxy/gerrit

The gerrit docker container shows that it couldn't contact the pg-gerrit on TCP:

steve@klaatu:~$ sudo docker logs gerrit
First time initialize gerrit...
[2016-02-18 20:57:27,566] [main] INFO com.google.gerrit.server.config.GerritServerConfigProvider : No /var/gerrit/review_site/etc/gerrit.config; assuming defaults
Generating SSH host key ... rsa(simple)... done
Initialized /var/gerrit/review_site

/gerrit-entrypoint.sh: ignoring /docker-entrypoint-init.d/*

Upgrading gerrit...
fatal: DbInjector failed
fatal: Unable to determine SqlDialect
fatal: caused by org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
fatal: caused by java.net.ConnectException: Connection refused

pg-gerrit does have an open port on 5432:
steve@klaatu:~$ sudo docker exec -it pg-gerrit bash
root@3d9c7c520637:/# cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:1538 00000000:0000 0A 00000000:00000000 00:00000000 00000000 999 0 2067181 1 0000000000000000 100 0 0 10 0

I noticed that pg-gerrit's log shows that it bounces the database during initialization, but this change to start up script has no effect on the ultimate outcome.

steve@klaatu:~/gerrit-docker (master)$ git diff
diff --git a/createGerrit.sh b/createGerrit.sh
index 879ae9e..574d135 100755
--- a/createGerrit.sh
+++ b/createGerrit.sh
@@ -23,7 +23,7 @@ docker run
-e POSTGRES_DB=reviewdb
-d ${POSTGRES_IMAGE}

-while [ -z "$(docker logs ${PG_GERRIT_NAME} 2>&1 | grep 'autovacuum launcher started')" ]; do
+while [ "$(docker logs ${PG_GERRIT_NAME} 2>&1 | grep 'autovacuum launcher started' | wc -l )" -ge 2 ]; do
echo "Waiting postgres ready."
sleep 1
done

Not sure how to proceed.

[thinkernel]

FATA[0001] Error response from daemon: Cannot start container 936f9931e733f30457269055785653bda80b4dd1d9c650d771d117c688e11a2f: Cannot link to a non running container: /gerrit AS /proxy/gerrit

It looks like you were using openfrontier/ci project to create the whole ci system, because this error shows proxy container cannot find runing gerrit container.
As you showed in sudo docker logs gerrit, gerrit container seems couldn't connect to the pg-gerrit container. If your postgres container is running properly, the 5432 port maybe blocked by your firewall. Can you put the output of docker logs pg-gerrit here?

btw. I update the gerrit & postgres image to the latest and chekout the latest source of ci project and gerrit-docker project. I cannot reproduce this issue in my environment.

[s-friedman]

Yes, I did a /ci/createContainer.sh. (Note that "/ci/createContainer.sh; ~/ci/start.sh" succeeds as pg-gerrit is ready.)

As requested, here are the pg-gerrit logs:

steve@klaatu:~$ sudo docker logs pg-gerrit
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data -l logfile start

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....LOG: database system was shut down at 2016-02-19 15:49:06 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
done
server started
CREATE DATABASE

CREATE ROLE

/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG: received fast shutdown request
LOG: aborting any active transactions
waiting for server to shut down....LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
done
server stopped

PostgreSQL init process complete; ready for start up.

LOG: database system was shut down at 2016-02-19 15:49:34 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started

[thinkernel]

Did you figure out this problem? The postgres container looks ok. You can start another postgres container as a client to connect this container. You can try command line like this one:
docker run -it --link pg-gerrit:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -d reviewdb -U gerrit2' The password is gerrit.
If this works, I really have no idea why the gerrit couldn't connect to the pg-gerrit.

[s-friedman]

Neither do I. Strangely, running "docker start gerrit" a couple of seconds later works -- it looks like a race condition in determining when pg-gerrit is up.

[s-friedman]

(I meant ~/ci/run.sh)

[thinkernel]

Did you change the values of HOST_NAME & LDAP_SERVER in the ci/config file to your actual host IP or FQDN?

[JackTan-TT]

Dear thinkernel,

I've encoutered the same issue while runing ~/ci/run.sh. It stucks in "Waiting postgres ready.".

bc2aaed03f2011d2716ce4c6024f6ff139e15a5bbddce3b113d2acdb7b75f61f
Waiting postgres ready.
Waiting postgres ready.
Waiting postgres ready.
Waiting postgres ready.

And I have change the config file as below, can you please share some suggestions on it?

#!/bin/bash

Hostname, IP or FQDN.

HOST_NAME=192.168.199.101

LDAP server configuration

Comment these variables out if you already have a running ldap server.

If SLAPD_DOMAIN and SLAPD_PASSWORD is set,

LDAP_SERVER and LDAP_ACCOUNTBASE will be overwritten automatically.

SLAPD_DOMAIN=example.com
SLAPD_PASSWORD=secret

LDAP_SERVER and LDAP_ACCOUNTBASE become unnecessary

if SLAPD_DOMAIN and SLAPD_PASSWORD are specified.

LDAP Server IP or FQDN.

LDAP_SERVER=ldap://192.168.199.101

LDAP account baseDN

LDAP_ACCOUNTBASE=ou=accounts,dc=example,dc=com

Gerrit administrator's uid in LDAP

GERRIT_ADMIN_UID=admin
GERRIT_ADMIN_PWD=passwd
GERRIT_ADMIN_EMAIL=admin@${SLAPD_DOMAIN}

A Nexus server will be started by default.

Specify this variable only if you want to use

an existing nexus server instead.

Jenkins uses NEXUS_WEBURL to setup maven repository mirror.

#NEXUS_WEBURL=http://192.168.199.101/nexus

[JackTan-TT]

After viewing the logs of pg-gerrit container, I found the below errors, my I have some suggestions?

2020-04-18 02:54:00.655 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
server stopped

PostgreSQL init process complete; ready for start up.

2020-04-18 02:54:00.667 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2020-04-18 02:54:00.667 UTC [1] LOG: listening on IPv6 address "::", port 5432
2020-04-18 02:54:00.990 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-18 02:54:01.294 UTC [66] LOG: database system was shut down at 2020-04-18 02:54:00 UTC
2020-04-18 02:54:01.428 UTC [1] LOG: database system is ready to accept connections
2020-04-18 02:56:22.709 UTC [75] LOG: invalid length of startup packet
2020-04-18 02:56:22.712 UTC [76] LOG: invalid length of startup packet
2020-04-18 02:56:22.716 UTC [77] LOG: invalid length of startup packet