Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
a5fa55c
consistently use print as a function
mvexel Oct 29, 2016
8c85a06
switching to requests in favor of urllib
mvexel Oct 29, 2016
36aa720
adding requirements.txt
mvexel Oct 29, 2016
a347a89
finishing up py3 compatibility
mvexel Oct 29, 2016
0872061
add bz2file to requirements
mvexel Oct 29, 2016
204c710
added print function from __future__ for Python 2 backwards compatibi…
mvexel Oct 29, 2016
657cb36
Merge remote-tracking branch 'upstream/master'
mvexel Sep 22, 2017
3b2ff2a
updating requirements
mvexel Jul 10, 2018
5a85bf1
py3 optimizations
mvexel Mar 7, 2019
147ed8b
http --> https
Mar 15, 2019
29364a5
http --> https
Mar 15, 2019
7b9a0e0
Merge branch 'master' of github.com:mvexel/ChangesetMD
Mar 15, 2019
b7e7ea8
speed up import with pyscopg2.extras.execute_batch
jlevente May 30, 2019
864024a
print status every 100k
jlevente May 30, 2019
72c1de1
fix type
Jul 31, 2019
42fb06d
delete extra line
Jul 31, 2019
3a25b95
fixed typp
Jul 31, 2019
38054a5
fixed typo
Aug 1, 2019
c5286f2
Merge remote-tracking branch 'jlevente/master'
mvexel Sep 3, 2019
a81cbd9
python 3 updates, organize imports
mvexel Sep 3, 2019
cac6cee
python 3 updates, organize imports, merging in improvements from jlev…
mvexel Sep 3, 2019
c706868
Merge branch 'master' of github.com:mvexel/ChangesetMD
mvexel Sep 3, 2019
79666d0
formatting, deprecating Python 2
mvexel Sep 11, 2022
77924b9
Create FUNDING.yml
mvexel Jan 30, 2023
0a370f9
Changed "yaml.load" to "yaml.full_load"
SomeoneElseOSM Jun 29, 2023
4142e0f
Updated README to contain a Debian summary section.
Jul 8, 2023
711fbc0
Consolidate readme changes from replication_changes_01 branch.
Oct 5, 2023
c1eed71
Merge pull request #2 from SomeoneElseOSM/Debian_12_01
mvexel Nov 13, 2023
ce41965
chores:
mvexel Nov 13, 2023
5fa2996
Merge pull request #3 from mvexel/maintenance
mvexel Nov 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# These are supported funding model platforms

github: [mvexel]
patreon: mvexel
ko_fi: mvexel
liberapay: mvexel
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
*.pyc
*.pyc
.DS_Store
venv/
__pycache__/
14 changes: 14 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Copyright (C) 2012 Toby Murray

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
44 changes: 30 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ It can also keep a database created with a weekly dump file up to date using min
Setup
------------

ChangesetMD works with python 2.7.
ChangesetMD works with Python 3.6 or newer.

Aside from postgresql, ChangesetMD depends on the python libraries psycopg2 and lxml.
On Debian-based systems this means installing the python-psycopg2 and python-lxml packages.

If you are using `pip` and `virtualenv`, you can install all dependencies with `pip install -r requirements.txt`.

If you want to parse the changeset file without first unzipping it, you will also need to install the [bz2file library](http://pypi.python.org/pypi/bz2file) since the built in bz2 library can not handle multi-stream bzip files.

For building geometries, ```postgis``` extension needs to be [installed](http://postgis.net/install).
Expand All @@ -26,21 +28,45 @@ It is easiest if your OS user has access to this database. I just created a user
createuser <username>


Full Debian build instructions
------------------------------

sudo apt install sudo screen locate git tar unzip wget bzip2 apache2 python3-psycopg2 python3-yaml libpq-dev postgresql postgresql-contrib postgis postgresql-15-postgis-3 postgresql-15-postgis-3-scripts net-tools curl python3-full gcc libpython3.11-dev libxml2-dev libxslt-dev

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

sudo -u postgres -i
createuser youruseraccount
createdb -E UTF8 -O youruseraccount changesets

psql
\c changesets
CREATE EXTENSION postgis;
ALTER TABLE geometry_columns OWNER TO youruseraccount;
ALTER TABLE spatial_ref_sys OWNER TO youruseraccount;
\q
exit


Execution
------------
The first time you run it, you will need to include the -c | --create option to create the table:

python changesetmd.py -d <database> -c
python changesetmd.py -d <database> -c -g

The `-g` | `--geometry` argument is optional and builds polygon geometries for changesets so that you can query which changesets were within which areas.

The create function can be combined with the file option to immediately parse a file.

To parse a dump file, use the -f | --file option.

python changesetmd.py -d <database> -f /tmp/changeset-latest.osm
python changesetmd.py -d <database> -g -f /tmp/discussions-latest.osm.bz2

If no other arguments are given, it will access postgres using the default settings of the postgres client, typically connecting on the unix socket as the current OS user. Use the ```--help``` argument to see optional arguments for connecting to postgres.

You can add the `-g` | `--geometry` option to build polygon geometries (the database also needs to be created with this option).
Again, the `-g` | `--geometry` argument is optional. Either of changeset-latest.osm.bz2 or discussions-latest.osm.bz2 or neither can be used to populate the database.

Replication
------------
Expand Down Expand Up @@ -122,13 +148,3 @@ Find all changesets that were created in Liberty Island:
SELECT count(id)
FROM osm_changeset c, (SELECT ST_SetSRID(ST_MakeEnvelope(-74.0474545,40.6884971,-74.0433990,40.6911817),4326) AS geom) s
WHERE ST_CoveredBy(c.geom, s.geom);

License
------------
Copyright (C) 2012 Toby Murray

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See the GNU Affero General Public License for more details: http://www.gnu.org/licenses/agpl.txt
Loading