How RocksDB + Xapian Power the New Game Object Explorer

[trieck]

🧩 How RocksDB + Xapian Power the New Game Object Explorer

In the latest release of BG3 Mod Studio (1.0.9), I added something I’ve wanted since the beginning:
a hybrid backend that combines RocksDB (structure) with Xapian (semantic full-text search) to power the new Game Object Explorer.

It turned out to be one of my favorite parts of the entire project.

---

🗄️ RocksDB → The Structural Backbone

RocksDB stores a complete object graph of the game using constant-time key/value lookup.

• root: → all top-level object categories
• child:<UUID> → prefix scan returns all children
• parent:<UUID> → enables backtracking to reconstruct hierarchy
• O(1) object access
• The DB itself encodes the full hierarchy

This means the tree control can expand instantly, even with tens of thousands of objects.

No global scans.
No loading the entire graph.
No slow traversal.

Just direct structural access.

---

🔍 Xapian → The Semantic Layer

Xapian handles full-text search across all objects.

With a boolean term like:
TYPE:GameObjects

the search automatically filters down to valid game objects.

Xapian provides:

• tokenization
• stemming
• ranking
• free-text matching
• extremely fast queries over large datasets

Importantly: Xapian does not know anything about the hierarchy.

It only knows which objects match the query.

---

🧠 Combining Them

When no search is active:

1. RocksDB returns all root: objects
2. The tree expands children using child:<UUID> prefix scans
3. Everything is structural and fast

This is the “hierarchy-first” view.

---

When a search query is active

1. We search Xapian using the user’s query.

2. From the top 1,000 search results, we extract the unique set of root types
   (e.g., character, item, scenery, etc.).
   These determine which top-level graph branches will appear.

3. We populate the tree with these root-type nodes.
   Each search result carries its own type, so we know exactly which root
   categories must be created.
   These root-type nodes serve as the attachment points for all reconstructed
   hierarchy paths.

4. For each search result, fetch the corresponding RocksDB object via its UUID key.

5. For each RocksDB object, iteratively collect all of its parents
   by following the parent:<UUID> chain until no parent remains.
   This produces an ordered list from leaf → parent → ... → root-type.

6. Iterate the parent list in reverse (root-type downward), and for each node:

   • Insert the node into the tree only if it does not already exist.
   • When the first parent encountered is a root-type, it is attached directly
     under the corresponding root-type node.
   • Subsequent parents attach under the previously inserted parent node.
     This reconstructs the complete hierarchy path.

7. Finally, insert the actual search result object into the tree
   under its last existing parent node.

---

🌱 Semantic Matches Can Produce Non-Semantic Children

Once the search results are obtained, the Explorer treats each matched object as an
expandable node within the hierarchy. This has an important consequence:

A search match may have children that do not themselves match the search
query in any way.

These children appear because:

• the matched object is a valid parent in the object graph, and
• the hierarchy allows expansion from any existing node.

This means:

• A node may appear because it directly matched the query (semantic match).
• Its ancestors may appear because they are structurally required (contextual match).
• Its children may appear because the matched node is expandable (structural continuation).

The resulting tree therefore contains:

1. Semantic leaves — objects returned by Xapian.
2. Contextual ancestors — required to place the leaves in the hierarchy.
3. Structural descendants — objects that exist under a matched node even if
   they contain no relevant keywords.

This produces a search experience that preserves both meaning and structure:
you get the objects you asked for, plus the surrounding context required to
navigate them naturally within the game’s data model.

---

✨ Why This Architecture Works Well

The combination of RocksDB and Xapian provides a clear separation of concerns:

• RocksDB handles the structural hierarchy of the object graph with fast key lookups and prefix scans.
• Xapian provides semantic full-text search across the entire dataset.
• The Explorer merges these two sources of information to produce a filtered, hierarchy-accurate view.

This design allows the system to remain fast, scalable, and easy to reason about.
RocksDB supplies the precise parent/child relationships, while Xapian identifies the relevant objects based on content.
Together, they produce a responsive and intuitive interface for navigating large game data sets.