.bag2 files?

[jhurliman]

Description

What do people think about introducing the option of a single file container for ROS2 bags? I'm thinking something like a tar file with the extension .bag2 since compression is handled at a lower level, but I mostly just want to start the discussion and get a temperature read.

The practical use cases are around storage and sharing. Uploading a single file for cloud storage can be an atomic operation (it's either uploaded or not, downstream processing actions can be triggered off the upload successfully completing) and verifying the upload succeeded is a more straightforward hash check of a single file where uploading a folder gets more into rsync territory. When sharing files with coworkers or on the Internet, it's convenient to post a single file for download instead of figuring out how to share a folder full of files (and confirm that you're received all of the files on the receiving end).

[danthony06]

Since no one else is jumping in, I'll add my 2 cents. Since we're still mostly on ROS 1, we typically fragment the bags according to file size while recording them, so during a field experiment we usually limit a bag to a couple of gigs. Our rationale is:

1. If the system crashes, it reduces our chances of losing all of the data from a trial, or having to do some kind of file recovery to fix any issues with an unexpected crash.
2. Since we're frequently in areas with poor network connectivity, it gives us a way to deliver a smaller amount of information back to our home location without needing to postprocess and filter a bag in the field.
3. Large ROS1 bags are generally painful to work with because of how the file access works. Principally, I think it does linear reads, so if I end up with a 60 gig bag file, opening and getting to that last gig of information is not a lot of fun.

I think ROS2's switch to sqlite on the backend makes point 3 less important, but the first two points still stand. I do think the single container helps when postprocessing, because it's a lot more convenient to pass in a single large file to an analysis or visualization script. It's also a lot more convenient for sharing, as you pointed out.

[emersonknapp]

I'm not opposed to this idea in principle - though I'd want to make sure we get the user experience right without complicating the options too much, and still support the other format.

The sqlite3 databases already encode the contents of the metadata.yaml file for a given file, so theoretically every .db3 file is independently usable, though right now the Reader APIs expect an aggregated metadata file to read. We could easily add an option to fall-back to the storage implementation's understanding of the metadata, as long as we make clear the expectation that a storage implementation must encode the metadata.

[danthony06]

Agreed. I think I'd want at least these two options.

1. At record time, pass a command line argument like ros2 bag record --monolith ...
2. As a post processing step, something like ros2 bag consolidate <path_to_bags>

The rationale for the first one is I can see users wanting a simple command to get this behavior at recording time. The second case would be for something like my usage pattern, where I have my field data fragmented into a bunch of files, but want an easy way to upload it to a colleague or simplify post-processing and analysis of it.

Is it possible to get this kind of behavior with our current set of command line arguments and utilities?

[chaoflow]

The sqlite3 databases already encode the contents of the metadata.yaml file for a given file, so theoretically every .db3 file is independently usable, ...

@emersonknapp Does this mean that metadata.yaml is purely redundant and all information to be stored in the db3 files? If yes, we need to take this into account in #782 as well.

[MichaelOrlov]

@chaoflow As far as I am understanding concept behind metadata.yaml is that it's playing role to provide easy accessible, human readable information about stored metadata without access to the databases via SQL requests.
Main purpose of metadata.yaml is to duplicate metadata from db3 file but not to store it in the first place.

It's comfy when you deal with multiple recordings and want to know what inside of these recordings without loading full stack, - you can simply open metadata.yaml in text editor.

[Karsten1987]

I see the argument of sharing (uploading/downloading) a bag file and thus have a single file containing all the info. I would say no to making it a single file by default for reasons being mentioned in this thread here already - i.e. composing various individual bag files via the metadata.yaml.
However, I could see this fit quite nicely in an additional verb to ros2 bag, something in the style of ros2 bag package <path/to/metadata.yaml>. That way a user can explicitly pack all related files into a common file and adapt the metadata.yaml file accordingly. tar seems like an obvious choice here, but keep in mind that we want to support the feature on all tier 1 platforms - Windows being one of them.

[amacneil]

I only just came across this thread, so cross referencing it with #855 since some similar discussions are happening over there.

[Flova]

Quite some time since the last comment, but I think that the new way of having a folder is quite painful to use. It is not that bad, but another minor inconvenience in ROS2. Many file formats (e.g. epub, apk, ...) consist of an internal folder structure that is hidden by a tar or zip archive. Accessing the metadata is quite easy as one just needs to grab an archive tool of his choice and view the metadata files. In rosbag2 it is the other way around. Everybody needs to grab such a tool if the bag should be transferred/downloaded/accessed. All of this just for the sake of easier access to the metadata. I hope nobody takes that personally, overall I think that e.g. sqlite is a great choice for the backend and appreciate the work that went into this. This is just some feedback from somebody who uses ros bags a lot and is not that involved in the development.

[jhurliman]

The MCAP storage plugin (https://github.com/ros-tooling/rosbag2_storage_mcap) is ready for testing. Once it has a few users that can confirm it's working for their day-to-day usage, we'll publish to rolling then humble as an apt package and close this issue out.

[emersonknapp]

This is related to work on #972 and #1074

[amacneil]

We took the ideas from this thread, #782, #855, and several others and incorporated them into the design of MCAP, a highly efficient, self-contained recording format for ROS 2 and more.

A ROS 2 MCAP storage plugin is available for Foxy, Galactic, Humble and Rolling, and using it is as simple as ros2 bag record -s mcap.

Therefore, I think we should close this issue now.

[danthony06]

Sorry if this is described somewhere, but I couldn't find it in the documentation or blog posts about this. Does MCAP rely on having the message definitions locally for embedding them in the file? In other words, if I attach my computer to a ROS2 system and I don't have the message definitions available on my computer, can I still record using MCAP and end up with a file with those definitions embedded in it?

…

On Tue, Sep 13, 2022, 1:01 AM Adrian Macneil ***@***.***> wrote: We took the ideas from this thread, #855 <#855>, and several others and incorporated them into the design of MCAP <https://mcap.dev/>, a highly efficient, self-contained recording format for ROS 2 and more. A ROS 2 MCAP storage plugin <https://github.com/ros-tooling/rosbag2_storage_mcap> is available for Foxy, Galactic, Humble and Rolling, and using it is as simple as ros2 bag record -s mcap. Therefore, I think we should close this issue now. — Reply to this email directly, view it on GitHub <#819 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBZWHFE6O3PMWYNQS72G6DV6AKE7ANCNFSM5ADXXCPA> . You are receiving this because you commented.Message ID: ***@***.***>