Duplicate node update crashes with --simplify-change #150
Description
When fetching today's daily updates from https://planet.osm.org/, I encountered the following error:
Aug 06, 2024 8:29:20 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.48.3
Aug 06, 2024 8:29:20 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Aug 06, 2024 8:29:20 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Aug 06, 2024 8:29:20 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Aug 06, 2024 8:29:20 AM org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader runImpl
INFO: Reading current server state. [ReplicationState(timestamp=Tue Aug 06 00:00:00 UTC 2024, sequenceNumber=4346)]
Aug 06, 2024 8:29:38 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-replication-interval failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are not sorted, previous entity type=Node, id=10767916505, version=21 current entity type=Node, id=10767916505, version=21.
at org.openstreetmap.osmosis.core.sort.v0_6.SortedHistoryChangePipeValidator.process(SortedHistoryChangePipeValidator.java:66)
at org.openstreetmap.osmosis.set.v0_6.ChangeSimplifier.process(ChangeSimplifier.java:50)
at org.openstreetmap.osmosis.core.sort.v0_6.ChangeSorter.complete(ChangeSorter.java:69)
at org.openstreetmap.osmosis.replication.v0_6.ReplicationDownloader.processComplete(ReplicationDownloader.java:118)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.runImpl(BaseReplicationDownloader.java:302)
at org.openstreetmap.osmosis.replication.v0_6.BaseReplicationDownloader.run(BaseReplicationDownloader.java:372)
at java.base/java.lang.Thread.run(Thread.java:840)
Aug 06, 2024 8:29:38 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:322)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:235)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:407)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:348)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Today's update file https://planet.osm.org/replication/day/000/004/346.osc.gz indeed shows a duplicate entry for the given node with ID 10767916505. Further inspection of the change files revealed the following. An oddity in the following two minutely updates seem to be the root cause:
https://planet.osm.org/replication/minute/006/205/197.osc.gz
https://planet.osm.org/replication/minute/006/205/198.osc.gz
197 contains just one change (exactly the one mentioned in my error message, node 10767916505), which is identical to a change in the 198 update. It is neither the first nor the last update in 198.
If I understand the workings of planet.osm.org correctly, the minutely updates are merged into hourly ones, which are in turn merged into the daily ones. See also https://wiki.openstreetmap.org/wiki/Planet.osm#Update_frequency. Somewhere in the merging of these files, these duplicates are concatenated to appear after each other in the daily update file.
Now, while this is technically speaking not an issue in Osmosis, but rather in the minutely updates, I think Osmosis should deal with this better to make it more robust. I forsee either of two solutions:
- Exact doubles should be merged (or rather, one should be discarded) when minutely updates are merged into hourly ones.
- The --simplify-change options should accept doubles, and simply take the last entry like it does with non-duplicates (This may be achieved by replacing the
>=with a>here , but perhaps I overlook some side-effects)