Support for GPX 1.1?

[cr]

We currently use gpxpy for handling GPX files. Currently only GPX 1.0 is supported by hxtool. GPX 1.0 defines a fixed set of XML nodes with a fixed set of properties to cover the most widespread use cases. So far these cover well what we need to do in hxtool.

GPX 1.1 is a whole different story. It only implements a subset of GPX 1.0 which does not cover all of the data we can extract from an HX870/890 GPS log. For that, GPX 1.1 implements extensions which we could theoretically use for fully covering our log data export. However, those extensions are more or less free form and different GPX 1.1 consumers expect different property names and formats while GPX 1.1 producers are all over the place.

This makes GPX 1.1 a de-facto broken standard, as every big vendor uses it a bit differently and that makes them incompatble amongst each other. hxtool would end up targeting only one specific log consumer or having to support a multitude. Also gpxpy would not let me add the extensions I wanted last time I looked.

So I think that unless the GPX 1.1 standard finally, well, standardizes – and I don't expect this to happen, as major vendors revel in the user lock-in the current situation creates – we should remain with GPX 1.0 as long as it enjoys the universal support it currently does.

[cr]

@johannessen, I think this thread might be interesting to you.

[johannessen]

While GPX certainly is nasty (as is most XML), I’m not sure I agree that 1.1 is broken. But I’d say the point is moot: Those parts of GPX that we really need are identical in 1.0 and 1.1.

Speed and heading data as reported by the HX870 is not useful and should not be included in the GPX export at all. If you still want to include this data, it likely doesn’t matter much if you choose 1.0 or a 1.1 extension, because most GPX consumers won’t read it either way. Going with 1.0’s <speed> and <course> elements would then seem to be the simplest solution.

Still, my recommendation would be to not pass on speed or heading data. At least not until a clear use case can be demonstrated that is not better served with having the GPX consumer compute average speed and course values from adjacent track points.

[johannessen]

Since you mention GPX issues: One issue that hasn’t been discussed are route points. The HX870 doesn’t really have those. Instead, it composes routes by storing references to waypoints.

Most contemporary GPS devices allow for many different kinds of route points. Besides regular waypoints, there might be long pre-defined lists with points of interest such as public buildings or transportation hubs, which can be used for routes. Frequently, using map features directly as route points is also possible. None of these positions can easily be stored in GPX, so route points are typically copied by value on those devices.

The GPX format doesn’t support referencing waypoints from within routes. This is not a limitation for most GPS devices, and in fact some use GPX as their native format for storing waypoints and routes.

But with the HX870, is is not possible to exactly represent the device configuration in GPX.

This is only a problem when one tries to flash the GPX file back to the radio. Unless special precautions are taken, duplicate waypoints will then be created for those waypoints that are used in routes.

One option to deal with this would be to introduce our very own proprietary GPX 1.1 extension. Sounds bad, I know, but it’s exactly what those are for: Extensions handle cases of no general interest, while keeping the standardised data (<rtept>, in this case) intact for other consumers.

Other options include abusing the <cmt> element (maybe put a random UUID in there to help resolving duplicates) or to try and detect duplicate positions by running a deep comparison on all waypoints and routepoints.

For now, I propose not handling this case at all until we have some experience regarding how useful GPX data in fact is when used with the HX870. After all, no sense in developing features that no one really needs…

[cr]

Before I go into detail for the specifics of GPX, there's a philosophical question here:

What is hxtool?

In my view it is a minimalist shim between the hardware on one side, and the user and other software on the other. On data export it is supposed to interpret the data from the device, but never filter. It should be up to the user to decide whether or not the speed and bearing data is useful, not the software.

On the import side for tracks and waypoints my approach would be to point users to one or two commonly used open source or online tools that can edit waypoints and routes, and then import their format. Trying to support a range of navigation software and workflows would result in high import code complexity and maintenance hell whenever anything changes with upstream tooling.

[johannessen]

That’s a reasonable point.

In the end, the weighing of practical or usability arguments with ones that concern the project’s philosophy is something only you can do. It’s your project, so your view is the one that should matter here.

(For clarity, I should perhaps add that absolutely no judgements are made here. I believe either point of view is perfectly valid. It’s simply a question of where you want to take this project, and it goes without saying that whatever you decide is fine by me.)

[cr]

While I invite any discussion about purpose and aims of hxtool, I'm just a burned child in terms of having to deal with the fallout of feature creep and code complexity on a daily basis, so for the time being I tend to stick with simple tools that do simple things, and do them well.

[cr]

I am mainly just expressing this because the last thing I want to see happening again is you dedicating your time to a big functionality patch and then me not liking it because it feels like the wrong direction. I mean, you can always continue with your fork, but I'd defintely prefer to work on this together and am grateful for every contribution.

[johannessen]

No worries :) The circumstances back then were very different, and I don’t see them occurring again with hxtool. From my point of view, there are no obstacles to working together here.

Also, simple tools are great!

[cr]

I just realized that I rushedly merged the dev branch to master that has the experimental GPX 1.1 support and hence does not export heading and speed. Where's my facepalm emoji?

[cr]

So how to move forward with GPX 1.1? It's an interesting idea to just use our own, proprietary GPX extension for hxtool. I'm worried that, as you outlined, it will ultimately condemn the extension data to "everybody just ignores it anyway" land. At least it would be a record of it, so later conversion would remain an option.

I'd slightly favor using a widely-supported standard extension, though. Garmin has a wide range of properties in their extensions that cover speed and bearing, so many in fact, that we might need to choose among a range of potential candidates.

I tried and failed adding Garmin-style extension to gpxpy which is very poorly documented and suffers from a well-pronounced undecipherable source code syndrome. Perhaps someone more skilled than myself can make sense of how to add those ElementTree DOM objects the gpxpy README mentions.

[johannessen]

So how to move forward with GPX 1.1?

Given that both of us seem to have expressed a preference of 1.0 over 1.1 at the top of this thread, I guess the answer would simply be not to move forward with 1.1. Or am I missing something here?

I tried and failed adding Garmin-style extension to gpxpy which is very poorly documented and suffers from a well-pronounced undecipherable source code syndrome. Perhaps someone more skilled than myself can make sense of how to add those ElementTree DOM objects the gpxpy README mentions.

While I’m certainly not more skilled than you when it comes to Python, I suppose I could give it a try anyway (if 1.1 really is the way forward, that is).