C bitfields are inherently non-interoperable

[ZimM-LostPolygon]

Dear ImGui uses C bit fields in some structs. While most of these fields are in imgui_internal.h and/or not particularly useful, this does technically mean the layout of those structs is completely compiler and/or platform defined:

• Different compilers might pack bitfields differently.
• Depending on platform endianness, compiler might pack bitfields in the opposite order.
• Pretty sure padding between bitfields is undefined.
• Don't get me started on alignment either.

From https://en.cppreference.com/w/cpp/language/bit_field.html:

The following properties of bit-fields are implementation-defined:

...

• Everything about the actual allocation details of bit-fields within the class object.
  • For example, on some platforms, bit-fields don't straddle bytes, on others they do.
  • Also, on some platforms, bit-fields are packed left-to-right, on others right-to-left.

GCC just goes "Determined by ABI" for pretty much everything:

https://gcc.gnu.org/onlinedocs/gcc/Structures-unions-enumerations-and-bit-fields-implementation.html

What this means is that those structs technically can't be represented in any other language other than C/C++. Yes, dear_bindings can understand bitfields and output them as part of metadata, but what's next? How would that metadata be used? If one were to, say, write a C# wrapper using dear_bindings, they'd have to reverse engineer and then replicate the actual memory layout of the data inside the struct, and even then, that would still be dependent on the compiler that built the native library. So either way, if a struct includes bitfields, it's completely non-interoperable with any other language. Annoyingly, even if one doesn't intend to ever touch those specific fields, they change how other fields are laid out in memory, including those of actual interest.

I've been thinking about this for quite a while, and other than going "YOLO, in real world all compilers would probably use the same layout", I can't think of any easy way to handle this. Here's some ideas:

• If a struct has any bitfields, it must be treated as opaque pointer, and any and all access to the fields is done through get/set accessors that dear_bindings will generate. Pretty terrible tbh, and introduces a whole new paradigm because of something pretty much no one will ever need.
• Remove bit size specifiers from dear imgui code itself, effectively converting them to regular fields. Very easy to do, but probably won't happen due to significantly increased memory usage.
• In dear imgui code, rework all bitfields to be just regular fields, and then do all the bit manipulations in dear imgui code manually. This would make the memory layout consistent and interopeable while keeping the memory savings, but would probably make that code quite messy...

**All bitfields as of 1.92.x**

(public)

ImFontBaked.MetricsTotalSurface:26 (uint)
ImFontBaked.WantDestroy:1 (uint)
ImFontBaked.LockLoadingFallback:1 (uint)

ImFontGlyph.Colored:1 (uint)
ImFontGlyph.Visible:1 (uint)
ImFontGlyph.SourceIdx:4 (uint)
ImFontGlyph.Codepoint:26 (uint)

(internal)

ImGuiTableColumn.SortDirection:2 (ImU8)
ImGuiTableColumn.SortDirectionsAvailCount:2 (ImU8)
ImGuiTableColumn.SortDirectionsAvailMask:4 (ImU8)

ImGuiStyleVarInfo.Count:8 (ImU32)
ImGuiStyleVarInfo.DataType:8 (ImGuiDataType)
ImGuiStyleVarInfo.Offset:16 (ImU32)

ImGuiWindow.SetWindowPosAllowFlags:8 (ImGuiCond)
ImGuiWindow.SetWindowSizeAllowFlags:8 (ImGuiCond)
ImGuiWindow.SetWindowCollapsedAllowFlags:8 (ImGuiCond)
ImGuiWindow.SetWindowDockAllowFlags:8 (ImGuiCond)
ImGuiWindow.DockIsActive:1 (bool)
ImGuiWindow.DockNodeIsVisible:1 (bool)
ImGuiWindow.DockTabIsVisible:1 (bool)
ImGuiWindow.DockTabWantClose:1 (bool)

ImGuiDockNode.AuthorityForPos:3 (ImGuiDataAuthority)
ImGuiDockNode.AuthorityForSize:3 (ImGuiDataAuthority)
ImGuiDockNode.AuthorityForViewport:3 (ImGuiDataAuthority)
ImGuiDockNode.IsVisible:1 (bool)
ImGuiDockNode.IsFocused:1 (bool)
ImGuiDockNode.IsBgDrawnThisFrame:1 (bool)
ImGuiDockNode.HasCloseButton:1 (bool)
ImGuiDockNode.HasWindowMenuButton:1 (bool)
ImGuiDockNode.HasCentralNodeChild:1 (bool)
ImGuiDockNode.WantCloseAll:1 (bool)
ImGuiDockNode.WantLockSizeOnce:1 (bool)
ImGuiDockNode.WantMouseMove:1 (bool)
ImGuiDockNode.WantHiddenTabBarUpdate:1 (bool)
ImGuiDockNode.WantHiddenTabBarToggle:1 (bool)

ImGuiTable.RowFlags:16 (ImGuiTableRowFlags)
ImGuiTable.LastRowFlags:16 (ImGuiTableRowFlags)

ImGuiStackLevelInfo.DataType:8 (ImGuiDataType)

ImGuiBoxSelectState.KeyMods:16 (ImGuiKeyChord)

ImGuiTableColumnSettings.SortDirection:2 (ImU8)
ImGuiTableColumnSettings.IsEnabled:2 (ImS8)
ImGuiTableColumnSettings.IsStretch:1 (ImU8)

ImFontAtlasRectEntry.TargetIndex:20 (int)
ImFontAtlasRectEntry.Generation:10 (int)
ImFontAtlasRectEntry.IsUsed:1 (uint)

[ShironekoBen]

Yeah, this is one of the more messy corners of the "everything in C is implementation-defined!" awkwardness. I don't think there's a huge amount that it's possible to do here in the general case - as you say, if you follow the letter of the standard you basically end up at "we can't assume anything about those structures and so we have to wrap literally everything everything in accessors", which is technically doable but would be awful in practice for a whole lot of reasons.

And since Dear Bindings isn't actually targeting a specific compiler/platform, we can't do what some binding generators do and actually calculate layout information (or get the compiler to do it for us) as part of the metadata. A specific consumer of the metadata JSON could theoretically do that for their desired target, though - it wouldn't be too hard to write something that emitted suitable struct definitions and a blob of reflection code that printf()ed all the offsets and values with each bit set. But that's wildly out-of-scope for what we can do here.

So my gut feeling is that if there's a specific use-case of something like "compiler X produces bitfields that are packed backwards relative to language Y" (for example) it might be viable to produce some helpers that perform conversion to/from an idealised standard bitfield notation or whatever as a means to work around that issue. But the problem space is just so broad that trying to pre-emptively fix it in a broad sense feels like a hopeless task, sadly.

[ZimM-LostPolygon]

That's sort of the annoying bit... Pretty much everything in dear imgui seems to be carefully designed in a way that makes binary interoperability easy, but then there's just a tiny sprinkle of bitfields here and there that kind of ruin it :/ So ultimately, I think, this issue is better suited in the main imgui repo, maybe removing bitfields in one way or another is less of a big deal for @ocornut that I might imagine.

Right now, for my C# wrapper, I try at least make sure all the bitfield groups end up taking the correct amount of bytes, so then at least the overall struct size matches up. It is, however, very magical and doesn't feel reliable at all...