The dbmgr tool needs to be faster/more frugal with memory

[jkoshy]

A complete planet.osm dump currently holds of the order of a billion nodes, ninety million ways and just a bit less than a million relations, per the current statistics for the OSM database.

In order to be able to deal with a data set of this size in a reasonable amount of time, the db-mgr ingestion tool needs to be sped up considerably, and also made frugal in its memory consumption.

[jkoshy]

Python's cProfile module was used to profile an ingestion run.

A faster JSON decoder may help.

>>> p.sort_stats('time').print_stats(25)
Wed Apr 20 22:52:58 2011    prof.1

       606063949 function calls (538093040 primitive calls) in 2355.761 CPU seconds

   Ordered by: internal time
   List reduced from 569 to 25 due to restriction <25>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2894113/2423987  293.772    0.000 1037.676    0.000 /usr/lib/python2.6/json/decoder.py:162(JSONObject)
62802266/179064  252.699    0.000 1249.725    0.007 /usr/lib/python2.6/json/scanner.py:38(iterscan)
  1841908  236.037    0.000  236.037    0.000 {built-in method acquire}
    89532  191.165    0.002  191.165    0.002 {method 'sendall' of '_socket.socket' objects}
   186671  100.853    0.001  100.853    0.001 {method 'recv' of '_socket.socket' objects}
 33943035   86.064    0.000   86.064    0.000 {_json.scanstring}
4965746/89532   84.138    0.000 1247.784    0.014 /usr/lib/python2.6/json/decoder.py:206(JSONArray)
 12299462   83.452    0.000  131.961    0.000 /usr/lib/python2.6/json/decoder.py:60(JSONNumber)
  2423987   81.286    0.000   81.286    0.000 ./apiserver/osmelement.py:138(from_mapping)
   897109   73.387    0.000  143.122    0.000 ./dbmgr/dbm_input.py:39(_make_osm_iterator)
128843635   67.806    0.000   67.806    0.000 {built-in method end}
 50502797   52.652    0.000   52.652    0.000 {built-in method scanner}
   897497   50.951    0.000   94.286    0.000 ./dbmgr/dbm_stats.py:98(increment_stats)
   153426   50.850    0.000  212.364    0.001 ./datastore/lrucache.py:258(insert_slab)
 31401133   46.769    0.000   46.769    0.000 {built-in method match}
 11241805   45.127    0.000   91.058    0.000 /usr/lib/python2.6/json/decoder.py:152(JSONString)
  3330996   32.307    0.000   76.072    0.000 ./apiserver/osmelement.py:116(__init__)
   153426   32.257    0.000   48.994    0.000 ./datastore/lrucache.py:233(_remove_slab_items)
   387550   27.715    0.000   27.742    0.000 {map}
  4443187   27.343    0.000   27.343    0.000 ./datastore/slabutil.py:35(_make_numeric_slabkey)
 43465635   24.734    0.000   24.734    0.000 {getattr}
   632818   24.690    0.000   91.443    0.000 /usr/lib/python2.6/threading.py:116(acquire)
 43700595   24.624    0.000   24.624    0.000 {built-in method span}
34895320/34895075   17.521    0.000   17.521    0.000 {len}
   897108   17.395    0.000 2285.756    0.003 ./dbmgr/dbm_ops.py:50(add_element)