Error thrown at pelias elastic create: EHOSTUNREACH

[darmentrout]

I'm getting the following output when running the pelias elastic create command. I can reach that internal URL just fine in the browser as well as curl.

$ pelias elastic create
Creating central-ohio_schema_run ... done

--------------
 create index 
--------------

Elasticsearch ERROR: 2020-10-07T20:36:28Z
  Error: Request error, retrying
  GET http://elasticsearch:9200/ => connect EHOSTUNREACH 192.168.160.2:9200
      at Log.error (/code/pelias/schema/node_modules/elasticsearch/src/lib/log.js:239:56)
      at checkRespForFailure (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:298:18)
      at HttpConnector.<anonymous> (/code/pelias/schema/node_modules/elasticsearch/src/lib/connectors/http.js:171:7)
      at ClientRequest.wrapper (/code/pelias/schema/node_modules/lodash/lodash.js:4949:19)
      at ClientRequest.emit (events.js:223:5)
      at Socket.socketErrorListener (_http_client.js:415:9)
      at Socket.emit (events.js:223:5)
      at emitErrorNT (internal/streams/destroy.js:92:8)
      at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
      at processTicksAndRejections (internal/process/task_queues.js:81:21)

Elasticsearch WARNING: 2020-10-07T20:36:29Z
  Unable to revive connection: http://elasticsearch:9200/

Elasticsearch WARNING: 2020-10-07T20:36:29Z
  No living connections

NoConnections [Error]: No Living connections
    at sendReqWithConnection (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:266:15)
    at next (/code/pelias/schema/node_modules/elasticsearch/src/lib/connection_pool.js:243:7)
    at processTicksAndRejections (internal/process/task_queues.js:76:11) {
  message: 'No Living connections'
}
unsupported elasticsearch version. try: >=7.4.2

pelias-docker: current version
docker-ce 3:19.03.13-3.el8
container-selinux noarch 2:2.135.0-1.module+el8.2.1+6849+893e4f4a
containerd.io 1.3.7-3.1.el8
docker-ce-cli 1:19.03.13-3.el8
libcgroup 0.41-19.el8

[missinglink]

The error message at the bottom indicates a problem:

unsupported elasticsearch version. try: >=7.4.2

You're probably trying to run an old version of elasticsearch which is no longer supported.

[darmentrout]

@missinglink I'm using the version that ships in the Docker container. The elasticsearch itself isn't the issue. I can get a response from it when I use curl outside of the Pelias container.

$ curl 192.168.160.2:9200
{
  "name" : "aaa9ab3af13e",
  "cluster_name" : "pelias-dev",
  "cluster_uuid" : "JdmdHLc2Ru2oLL95EFL9LA",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[Asherslab]

I'm also getting the same error, freshly cloned repo, ran the exact commands from the README, same error as reported above

[missinglink]

Hmm ok thanks for the confirmation, I'll run an import locally and hopefully I can reproduce it and fix it.

[missinglink]

I'm not able to reproduce, can you please provide more information:

1. have you made any edits to the networks section of docker-compose.yml?
2. what is the output of docker network list?
3. what is the output of docker run --network=pelias_default --rm -it --entrypoint=curl pelias/api:master elasticsearch:9200 (with the pelias/elasticsearch:7.5.1 container running)?

[missinglink]

I'm on Mac, I'm assuming you're both on Linux?

[Asherslab]

am on centos 7.

however i noticed what i believe the issue is while attempting to get the information you asked for.
seems that elasticsearch didn't have permissions to the data directory and was erroring out.
i adjusted the permissions (lazying with a recursive 777) and it seems to be running now.
or at least got further than it did before

[missinglink]

agh ok, then I think you (@Asherslab) have a different issue as @darmentrout said he has elasticsearch running.

We're aware of the permission issue, it's being tracked in #74, its common when users use a root account (or sudo) to create the data dir. We are one of the few projects 'doing it properly' by running the docker containers as non-root users, so the docker user doesn't have permission to access directories created by root.

Regarding this issue, here is the script I used to test the bug report with the output from my machine:

#!/bin/bash

TMPDIR='/tmp/bugreport'
rm -rf "${TMPDIR}"
mkdir -p "${TMPDIR}"

cd "${TMPDIR}"
git clone git@github.com:pelias/docker.git && cd docker/projects/portland-metro

# configure data dir
mkdir ./data
if [[ "$OSTYPE" == "darwin"* ]]; then
  gsed -i '/DATA_DIR/d' .env
else
  sed -i '/DATA_DIR/d' .env
fi
echo 'DATA_DIR=./data' >> .env

# ensure everything is down
pelias compose down

# create index
pelias compose pull
pelias elastic start
pelias elastic wait
pelias elastic create

➜  /tmp ./bug.sh
Cloning into 'docker'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 1639 (delta 5), reused 11 (delta 4), pack-reused 1622
Receiving objects: 100% (1639/1639), 446.72 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (1045/1045), done.
Stopping pelias_elasticsearch ... done
Removing pelias_elasticsearch ... done
Removing network pelias_default
Pulling libpostal     ... done
Pulling schema        ... done
Pulling api           ... done
Pulling placeholder   ... done
Pulling whosonfirst   ... done
Pulling openstreetmap ... done
Pulling openaddresses ... done
Pulling transit       ... done
Pulling csv-importer  ... done
Pulling polylines     ... done
Pulling interpolation ... done
Pulling pip           ... done
Pulling elasticsearch ... done
Pulling fuzzy-tester  ... done
Creating network "pelias_default" with driver "bridge"
Creating pelias_elasticsearch ... done
waiting for elasticsearch service to come up
.........Elasticsearch up!
Creating pelias_schema_run ... done

--------------
 create index
--------------

[put mapping] 	 pelias { acknowledged: true, shards_acknowledged: true, index: 'pelias' }

[darmentrout]

@missinglink here is the output from the test script:

[armentroutdd@DEV-WEB-020 ~]$ cd test
[armentroutdd@DEV-WEB-020 test]$ ./test.sh
Cloning into 'docker'...
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 1674 (delta 8), reused 22 (delta 7), pack-reused 1624
Receiving objects: 100% (1674/1674), 452.29 KiB | 4.81 MiB/s, done.
Resolving deltas: 100% (1048/1048), done.
Removing network pelias_default
Pulling libpostal     ... done
Pulling schema        ... done
Pulling api           ... done
Pulling placeholder   ... done
Pulling whosonfirst   ... done
Pulling openstreetmap ... done
Pulling openaddresses ... done
Pulling transit       ... done
Pulling csv-importer  ... done
Pulling polylines     ... done
Pulling interpolation ... done
Pulling pip           ... done
Pulling elasticsearch ... done
Pulling fuzzy-tester  ... done
Creating network "pelias_default" with driver "bridge"
Creating pelias_elasticsearch ... done
waiting for elasticsearch service to come up
..............................

Elasticsearch did not come up, check configuration
Creating pelias_schema_run ... done

--------------
 create index 
--------------

Elasticsearch ERROR: 2020-10-09T13:41:20Z
  Error: Request error, retrying
  GET http://elasticsearch:9200/ => connect EHOSTUNREACH 172.21.0.2:9200
      at Log.error (/code/pelias/schema/node_modules/elasticsearch/src/lib/log.js:239:56)
      at checkRespForFailure (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:298:18)
      at HttpConnector.<anonymous> (/code/pelias/schema/node_modules/elasticsearch/src/lib/connectors/http.js:171:7)
      at ClientRequest.wrapper (/code/pelias/schema/node_modules/lodash/lodash.js:4949:19)
      at ClientRequest.emit (events.js:223:5)
      at Socket.socketErrorListener (_http_client.js:415:9)
      at Socket.emit (events.js:223:5)
      at emitErrorNT (internal/streams/destroy.js:92:8)
      at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
      at processTicksAndRejections (internal/process/task_queues.js:81:21)

Elasticsearch WARNING: 2020-10-09T13:41:23Z
  Unable to revive connection: http://elasticsearch:9200/

Elasticsearch WARNING: 2020-10-09T13:41:23Z
  No living connections

NoConnections [Error]: No Living connections
    at sendReqWithConnection (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:266:15)
    at next (/code/pelias/schema/node_modules/elasticsearch/src/lib/connection_pool.js:243:7)
    at processTicksAndRejections (internal/process/task_queues.js:76:11) {
  message: 'No Living connections'
}
unsupported elasticsearch version. try: >=7.4.2

[missinglink]

This shows the error Elasticsearch did not come up, check configuration, but previously you said it was running?

[missinglink]

Please post the output of cat .env and id along with the commands mentioned in #219 (comment)

What are the folder permissions of the directory defined as DATA_DIR in your .env file?

➜  portland-metro git:(docker_permissions) ✗ cat .env
COMPOSE_PROJECT_NAME=pelias
DATA_DIR=/data/pelias/docker/projects/portland-metro

➜  portland-metro git:(docker_permissions) ✗ ls -lah /data/pelias/docker/projects/portland-metro
total 0
drwxr-xr-x  14 peter  staff   448B Jul 29 16:53 .
drwxr-xr-x   8 peter  staff   256B Oct  9 13:49 ..
drwxr-xr-x   2 peter  staff    64B Apr 24 14:11 blacklist
drwxr-xr-x   3 peter  staff    96B Apr 24 14:11 csv
drwxr-xr-x   3 peter  staff    96B Jul 30 15:54 elasticsearch
drwxr-xr-x  20 peter  staff   640B Apr 24 14:45 interpolation
drwxr-xr-x   5 peter  staff   160B Oct  7 11:57 openaddresses
drwxr-xr-x   3 peter  staff    96B Apr 24 14:11 openstreetmap
drwxr-xr-x   4 peter  staff   128B Apr 24 14:13 placeholder
drwxr-xr-x   3 peter  staff    96B Apr 24 14:13 polylines
drwxr-xr-x   3 peter  staff    96B Jul 30 13:46 spatial
drwxr-xr-x   3 peter  staff    96B Apr 24 14:11 tiger
drwxr-xr-x  16 peter  staff   512B Apr 24 14:12 transit
drwxr-xr-x   3 peter  staff    96B Apr 24 14:11 whosonfirst

➜  portland-metro git:(docker_permissions) ✗ ls -lah /data/pelias/docker/projects/portland-metro/elasticsearch
total 0
drwxr-xr-x   3 peter  staff    96B Jul 30 15:54 .
drwxr-xr-x  14 peter  staff   448B Jul 29 16:53 ..
drwxrwxr-x   3 peter  staff    96B Jul 30 15:54 nodes

[darmentrout]

$ cat .env
COMPOSE_PROJECT_NAME=pelias
DOCKER_USER=1000
DATA_DIR=./data

have you made any edits to the networks section of docker-compose.yml?

no

what is the output of docker network list?

$docker network list
NETWORK ID          NAME                   DRIVER              SCOPE
faa1e1381ef6        _default               bridge              local
6f778ed923eb        bridge                 bridge              local
f06a8e7ddc97        central-ohio_default   bridge              local
73a29a79390d        host                   host                local
0240b010d95a        none                   null                local
8b2f06553816        pelias_default         bridge              local

what is the output of docker run --network=pelias_default --rm -it --entrypoint=curl pelias/api:master elasticsearch:9200 (with the pelias/elasticsearch:7.5.1 container running)?
curl: (6) Could not resolve host: elasticsearch

$ pwd
/tmp/bugreport/docker/projects/portland-metro
$ ls -lah
total 24K
drwxrwxr-x.  6 armentroutdd armentroutdd  143 Oct  9 11:05 .
drwxrwxr-x. 18 armentroutdd armentroutdd 4.0K Oct  9 11:05 ..
drwxrwxr-x.  2 armentroutdd armentroutdd   21 Oct  9 11:05 blacklist
drwxrwxr-x.  3 armentroutdd armentroutdd   27 Oct  9 11:05 data
-rw-rw-r--.  1 armentroutdd armentroutdd 3.5K Oct  9 11:05 docker-compose.yml
-rw-rw-r--.  1 armentroutdd armentroutdd   61 Oct  9 11:05 .env
-rw-rw-r--.  1 armentroutdd armentroutdd 4.9K Oct  9 11:05 pelias.json
-rw-rw-r--.  1 armentroutdd armentroutdd 1.2K Oct  9 11:05 README.md
drwxrwxr-x.  2 armentroutdd armentroutdd   54 Oct  9 11:05 synonyms
drwxrwxr-x.  2 armentroutdd armentroutdd   87 Oct  9 11:05 test_cases

[missinglink]

can you please post the output of ls -lah data?

[missinglink]

and id