Lack of Elasticsearch documentation

[rdingwell]

There is a lack of information regarding how to setup Elasticsearch and the mongo-connector for use with the Matchminer set of tools. Notably missing is the versions of ES that are supported, any associated indexing schemas as well as commands for syncing mongo and ES.

Attempting to run a mongo-connector command to sync the 2 systems yielded errors in the log file for multiple type mappings for particular keys. This leads me to believe that a ES version of 5 or less is required as multiple type mappings are no longer supported in ES 6 and greater.

I understand that this may be provided in the docker images but we are attempting to evaluate building additional functionality into matchminer and need to set this up in a developer instance which is proving difficult with missing documentation.

[ethansiegl]

Hello. Yes you are correct that an elasticsearch version of 5 or earlier is required for the application to function correctly.

You are also correct that there are mappings, settings and analyzers required to load if you are using a vanilla elasticsearch instance. I can provide you here with links to some of the mapping, analyzer, and analysis files you need here, as well as a bash script you can use to load the files into the ES instance. You will also need to setup a local elasticsearch instance which uses a custom synonym file.

For reference, the docker container which has all of these settings, mappings, synonyms, and other configurations pre-loaded can be accessed here.

Elasticsearch Setup
mapping.json
analyzers.json
analysis.json
synonyms.txt
elasticsearch.yml
settings.json
elasticsearch_setup.sh

Once you have downloaded all of the files, first startup an elasticsearch instance in a custom network. You will need to update some directory paths. Here is a compose file for reference:

docker network create mm_custom_network

version: '3'
services:

  elastic:
    container_name: prod_elastic
    restart: "no"
    image: elasticsearch:2.4.0
    ports:
      - "9200:9200"
    volumes:
      - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./synonyms.txt:/usr/share/elasticsearch/config/synonyms.txt
      - /data/es_volume/data:/usr/share/elasticsearch/data    #update directory
      - /data/matchminer/staging/elastic/backup:/backup      #update directory 
    networks:
      - mm_custom_network

docker-compose up -d elastic

Then run the bash script:
./elasticsearch_setup.sh

Let me know if this helps. I will try and find some time to add this to the documentation

[rdingwell]

Thanks, I think there may still need to be a settings.json file that is not included in the list above. The script runs but with errors because it cannot find it.

[ethansiegl]

Sorry forgot that one. Here's the file:

settings.json

[rdingwell]

Thanks.
One other thing I noticed is that it's not just ES 5.x or earlier it's 5.2 or earlier in case anyone else is trying to set everything on their own