Next generation of rosdistro tooling?

[mikepurvis]

Not sure if there's a better place to be having this discussion, but there are a few ways in which python-rosdistro is starting to get pretty creaky for us. Rather than hacking around its limitations or building a bunch of proprietary parallel infrastructure, we're wondering if there would be any enthusiasm for a more ambitious project to replace it. Some particular goals would be:

• Excise codepaths that handle rosbuild packages, stacks, manifest.xml, and anything else pre-Groovy. None of this stuff has been relevant in years, and is probably rotted anyway.
• Drop Python 2 compatibility.
• Allow packages to be discovered and indexed in the cache without them needing to contain a package.xml file, by reusing the plugin infrastructure that's currently part of colcon. This would mean caching serialized PackageDescriptor instances rather than package.xml contents as strings.
• Provide a plugin mechanism to hitch additional data onto the distribution at the distribution.yaml layer. Example uses might include storing information about a staging or release branch, or about what the upstream url is when a repo has been forked. Having some flexibility here would also make it easier in the future to add something like test_pull_requests without needing a new REP.
• Also provide plugin hooks to add additional data at the cache layer. There are lots of possibilities for how this could be useful, but one would be computing and storing hashes of certain files or groups of files in order for some other process to detect when a particular action must be taken, such as rebuilding documentation or running an ABI checker when headers mutate.

Several of these ideas address pain points that there are other open tickets for, such as #77, #82, and ros-infrastructure/rosinstall_generator#33.

I believe everything proposed here could be done without actually changing anything about REPs 137, 141, or 143, and it's even conceivable that the existing interfaces exposed by rosdistro could be maintained, such that tools like ros_buildfarm, rosinstall_generator, and others would require only minimal porting.

[dirk-thomas]

Excise codepaths that handle rosbuild packages, stacks, manifest.xml, and anything else pre-Groovy. None of this stuff has been relevant in years, and is probably rotted anyway.

👍

Drop Python 2 compatibility.

I don't think that is feasible in a lot of cases where a generic Python package like this is released into for all platforms targeted by actively supported ROS distros. That would only be possible for a new tool which only offers Python 3 compatibility from day one (or all Python 2 platforms are aged out of support which would be ROS Melodic in 2023).

Allow packages to be discovered and indexed in the cache without them needing to contain a package.xml file, by reusing the plugin infrastructure that's currently part of colcon. This would mean caching serialized PackageDescriptor instances rather than package.xml contents as strings.

That sounds interesting. For some use cases like releasing packages with bloom these information wouldn't be sufficient at the moment. Not sure if the idea is to make the information complete or just not support blooming for these kind of packages.

Provide a plugin mechanism to hitch additional data onto the distribution at the distribution.yaml layer. Example uses might include storing information about a staging or release branch, or about what the upstream url is when a repo has been forked. Having some flexibility here would also make it easier in the future to add something like test_pull_requests without needing a new REP.

Do you have examples for what you would imagine this to be used for?

---

we're wondering if there would be any enthusiasm for a more ambitious project to replace it.

While I agree with you that the current system is aging and it would be good to work towards the next generation of this infrastructure I doubt that there is much resources available for such a push.

[mikepurvis]

We no longer use Bloom, so that wouldn't be a primary consideration for an effort of this kind that we were spear-heading. When the time came, the interested parties would have to see about the effort involved in bringing the missing information to PackageDescriptor.

Provide a plugin mechanism to hitch additional data onto the distribution at the distribution.yaml layer. Example uses might include storing information about a staging or release branch, or about what the upstream url is when a repo has been forked. Having some flexibility here would also make it easier in the future to add something like test_pull_requests without needing a new REP.

Do you have examples for what you would imagine this to be used for?

Well, I stated a few in the paragraph you quoted. :) But since that was written, you guys have also gone through the whole process again with REP 154 and the addition of test_abi. The big one for us is better managing our relationship to upstream repos. Having information about multiple origins and branches/tags right in the distribution.yaml would make it much easier for us to have tooling that quickly shows us how how far ahead or behind our upstreams we are.

In any case, it sounds like in the short term if we want to pursue this, we should do so by kicking off a new toolset with a new name, and in a few years when Melodic is done, we can look at what we have and see if it's something which would be of interest to the community to adopt.

[dirk-thomas]

In any case, it sounds like in the short term if we want to pursue this, we should do so by kicking off a new toolset with a new name, and in a few years when Melodic is done, we can look at what we have and see if it's something which would be of interest to the community to adopt.

Since the upcoming Noetic release is the last promised ROS 1 distro I doubt that anyone will spend effort on changing such an essential part for ROS 1 after that release is already out for that long. And if your only focus is ROS 2 you don't have to consider Python 2 anyway.