Postgres failed to initialize database

[minh10huy]

I'm encountering the following issue: m3fs checks the PostgreSQL connection and gets confused, but the PostgreSQL Docker container is still running. Could this error be caused by m3fs checking PostgreSQL while it's still starting up?

INFO[0001] Started postgresql container 3fs-postgres successfully NODE=open3fs-stor01 TASK=CreatePgClusterTask
ERRO[0001] [gorm] failed to initialize database, got error failed to connect to user=postgres database=open3fs: 192.168.1.241:5432 (192.168.1.241): dial error: dial tcp 192.168.1.241:5432: connect: connection refused
WARN[0001] Failed to create db connection to 192.168.1.241:5432: open database: failed to connect to user=postgres database=open3fs: 192.168.1.241:5432 (192.168.1.241): dial error: dial tcp 192.168.1.241:5432: connect: connection refused NODE=open3fs-stor01 TASK=CreatePgClusterTask

[apilighthouse]

I have successfully reproduced this issue. The root cause lies in the timing of m3fs's PostgreSQL connection check during initialization, combined with network configuration specifics.

Reproduction Environment:

On a Windows system, install Docker Desktop with WSL 2 backend enabled.
Install the official Ubuntu distribution from the Microsoft Store.
Configure Docker to be accessible within the Ubuntu WSL environment and set it as the m3fs master node.
In the m3fs configuration file, designate the same Ubuntu WSL instance as a worker node.
When m3fs is started within this WSL environment, it invokes Docker to launch the PostgreSQL container using the host network mode—meaning the container directly shares the host’s network stack.

However, here lies the problem: under WSL 2, Docker actually runs inside a lightweight virtual machine based on Hyper-V. When a container uses host networking, the term "host" refers to the Docker VM, not the WSL 2 instance or the Windows host itself.

This creates a network reachability issue. When m3fs running in WSL attempts to connect to PostgreSQL at an address like 192.168.1.241:5432, that IP may refer to an interface on the WSL or Windows host. However, due to the network isolation of the Docker VM and the use of host mode, this address cannot be correctly routed from within the WSL environment—resulting in the connection refused error.

It's important to note that Docker was originally designed for native Linux server environments, where the host OS, Docker daemon, and containers share a consistent and predictable network model. WSL 2, while a powerful tool for Linux compatibility on Windows, introduces an additional layer of virtualization and network abstraction that deviates from this traditional setup. As a relatively new technology, WSL 2 may not be fully compatible with all Docker features—especially advanced networking modes like host—which were not originally intended for such hybrid environments.

Recommended Solution:

The simplest and most effective workaround is to avoid using WSL environments as m3fs worker nodes. Instead, use a dedicated Linux virtual machine or a separate physical/cloud server as the worker node. This ensures a clear and predictable Docker networking environment, avoiding the complexities introduced by WSL 2's network virtualization.

Testing confirms that when a standalone Linux node is used as the worker, the PostgreSQL container starts correctly and m3fs can successfully establish a connection—resolving the issue.

[wbashiti]

I am facing a similar DB start issue that is preventing the cluster from starting.

Root Cause:
There is a race condition in the m3fs deployment tool:

The script starts the PostgreSQL container and immediately tries to connect, before the DB is ready.

This leads to connection refused errors, which cascade and cause dependent services (e.g., mgmtd, meta) to fail.

Diagnostic Evidence:

Log entries show Started postgresql container ... successfully followed by immediate connect: connection refused.

The database’s log (docker logs 3fs-postgres) shows "database system is ready to accept connections" several seconds after deployment tried to connect.

`INFO[0002] Started postgresql container 3fs-postgres successfully NODE=st04 TASK=CreatePgClusterTask

ERRO[0002] [gorm] failed to initialize database, got error failed to connect to user=postgres database=open3fs: 172.21.204.24:5432 (172.21.204.24): dial error: dial tcp 172.21.204.24:5432: connect: connection refused

WARN[0002] Failed to create db connection to 172.21.204.24:5432: open database: failed to connect to user=postgres database=open3fs: 172.21.204.24:5432 (172.21.204.24): dial error: dial tcp 172.21.204.24:5432: connect: connection refused NODE=st04 TASK=CreatePgClusterTask

INFO[0008] Running task CreateFdbClusterTask `