Build for other countries

[december1990]

hey there, I managed to build and test the portland-metro project successfully and now I want to build the project for other countries that aren't listed in projects folder, for example build for the great britain, how am I to do that ?

thanks

[missinglink]

Hi @december1990, please don't open multiple issues for the same thing (pelias/pelias#923).

You mentioned in your issue description that you've resolved #279, if that's the case please close the issue.

Regarding your question, you can copy one of the existing project directories and modify it to build any region.
There are some values which you'll need to change in the pelias.json file, but the docker-compose.yml file doesn't need to be modified.

in particular the imports.geonames.countryCode, imports.openstreetmap.download.sourceURL (and corresponding imports.openstreetmap. import.filename), import.openaddresses.files, whosonfirst.countryCode and whosonfirst.importPlace variables.

the documentation for this is sadly lacking at the moment but I will try to write a blog post about it to explain what all the settings are and how to configure them.

[JosephKuchar]

Hi,

I thought I'd piggyback on this issue instead of creating another. I'm planning to create a build for Canada, and I had a question about the best practices for incorporating openaddresses data. Several of the examples I've looked at seem to reference local downloads of the OA data as opposed to harvesting from a URL - is this the norm? Canada has something approaching 100 sources in OA, if I recall correctly, so I'm hoping there's an alternative to downloading them all individually.

Thanks,
Joseph

[missinglink]

Hi @JosephKuchar unfortunately the code for openaddresses downloads pre-dates the OA team having an API by several years.

It would be nice to modernize the code by having it call their API for a list of sources and then filter by prefix, or whatever.

I started writing something to do exactly that but then got busy with other things, you might be able to use my code to either scrape a list of sources or to open a PR to add support for it.

[missinglink]

const _ = require('lodash');
const OA = require('@openaddresses/lib');
const config = require('pelias-config');

class OpenAddressesAPI {
  constructor() {
    this.config = _.get(config.generate(), 'imports.openaddresses', {});
    this.client = new OA(_.pick(this.config, 'token'));
  }

  // remove file extensions from 'source'
  normalize(source) {
    if (!_.isString(source)) { return source; }
    return source.replace(/\.[^/.]+$/, '')
  }

  // return the http url for a specific job id
  url(job) {
    return `${this.client.url}/api/job/${job}/output/source.geojson.gz`
  }

  // if the 'validated' mode is enabled (for financial supporters only)
  isValidatedModeEnabled() {
    return _.get(this.config, 'validated') === true
  }

  async lookup() {
    // files listed in pelias.json
    const files = _.get(this.config, 'files', []);

    // lookup each source from the remote API
    return await Promise.all(files.map(async (filename) => {

      // normalize 'source' property
      // support the 'validated' property for financial supporters
      const payload = {
        source: this.normalize(filename),
        layer: 'addresses',
        validated: this.isValidatedModeEnabled()
      };

      // request extended info and return the first result
      const versions = await this.client.cmd('data', 'list', payload);
      return _.isArray(versions) && !_.isEmpty(versions) ? _.head(versions) : {};
    }));
  }
}

module.exports = OpenAddressesAPI

[JosephKuchar]

Thanks @missinglink ! I actually ended up writing a short Python script to loop through OpenAddresses's Canadian sources listed on their github, then query those to the OpenAddresses Data and Jobs APIs to collect all the data.

I'm encountering an issue now with the installation process: when I run
pelias compose pull
I get a permissions error:
File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 54, in _get_stream with io.open(self.dotenv_path, encoding=self.encoding) as stream: PermissionError: [Errno 13] Permission denied: './.env'

The drive it's being installed on is generally one that requires root permissions to make modifications to, but I gave myself ownership of the pelias folder and all of its contents, and even after explicitly setting the .env file's permissions to be read and write by everyone this error is persisting. Any ideas?

[missinglink]

pelias compose pull is just a thin wrapper over docker-compose pull.

What happens when you run the latter command? Presumably the same error?

[JosephKuchar]

Aha, thanks - that also failed, and I see now the issue is docker was installed with Snap, so I'll need to just reinstall it properly.

[december1990]

Hi

I want to build geocoding service for great britain, how can i get the right value(s) for whosonfirst.importPlace for great britain ? since pbf file for this area is too large I get an error when running "pelias prepare all" command, so polylines can't be created. do I need polylines and interpolation to use geocoding service ? if yes, after downloading prepared .csv file should I run prepare all again or I can go to the next command ?

[december1990]

Hi

I'm running Pelias for one of the projects, while it's running I try to build for another country but I think data is imported into the data/elasticsearch of the running project, is this the case ?? when building for new country should I stop and remove previous containers ? and I have another question when I run import does it remove previous data or appends it to the db ??

thanks

[missinglink]

It's possible to add new data to an existing project but it's much easier to run the build again.

I would suggest creating a new 'project' or editing your existing project and setting a new DATA_DIR and starting the build steps from the beginning.

[december1990]

1. do I need to stop the running project and start new build for the new project from the beginning and should I run elastic start, wait, create again for every new project ?

2. data is imported to the elasticsearch and is it saved for each project separately ? I guess although I was in a different project directory, data was imported to the running project's /data/elasticsearch

3. is this pelias,json correctly formatted ? specially for whosonfirst, I also create polylines out of pbf using docker-valhalla-baseimage.

{
"logger": {
"level": "debug",
"timestamp": false
},
"esclient": {
"apiVersion": "7.5",
"hosts": [
{ "host": "elasticsearch" }
]
},
"elasticsearch": {
"settings": {
"index": {
"refresh_interval": "10s",
"number_of_replicas": "0",
"number_of_shards": "1"
}
}
},
"api": {
"services": {
"pip": { "url": "http://pip:4200" },
"libpostal": { "url": "http://libpostal:4400" },
"placeholder": { "url": "http://placeholder:4100" },
"interpolation": { "url": "http://interpolation:4300" }
},
"defaultParameters": {
"focus.point.lat": 52.37,
"focus.point.lon": 4.89
}
},
"imports": {
"adminLookup": {
"enabled": true
},
"openstreetmap": {
"download": [
{ "sourceURL": "http://download.geofabrik.de/europe/britain-and-ireland-latest.osm.pbf" }
],
"leveldbpath": "/tmp",
"maxSizeBytes": 2361231613,
"datapath": "/data/openstreetmap",
"import": [{
"filename": "britain-and-ireland-latest.osm.pbf"
}]
},
"polyline": {
"datapath": "/data/polylines",
"files": [ "extract.0sv" ]
},
"whosonfirst": {
"datapath": "/data/whosonfirst",
"countryCode": ["GB", "IE"],
"importPostalcodes": true,
"importPlace": [85633159, 85633241]
}
}
}

[missinglink]

1. stop, change DATA_DIR and start again from the beginning of the instructions 2. yes this is relative to DATA_DIR. 3. yes, hard to tell but looks good

[ebdavison]

Thanks @missinglink ! I actually ended up writing a short Python script to loop through OpenAddresses's Canadian sources listed on their github, then query those to the OpenAddresses Data and Jobs APIs to collect all the data.

Would you mind sharing the python script you used? I could be helpful for other regions than just canada.