Key not found on NTDS

[Matthijsy]

During querying different NTDS files i get the following error:

dissect.database.ese.exception.KeyNotFoundError: Key not found: b'\x7f\x80\x00\x11y'

(the exact key is different every time, but always seems to start with \x7f\x80)

This happens for example when querying users, computers, group policies. This currently results in target-query erroring, and missing information. I also pointed this out in PR #38, and there @Schamper mentioned that this might be due to transactions logs. However, I can dump this information using the ntdissector, so that seems not to be the case.

Unfortonately this only happens on NTDS files I have which I cannot share. I tried to reproduce it on NTDS files available on VT, but the ones I found there did not yield this problem. If there is any debug log / stracktrace I could recreate which could help in debugging the cause please let me know!

[Schamper]

ntdissector does not do quite the same level of properly querying information as we do, so we're more prone to failures like this. If you already have any stacktraces available, we could check at least where it's failing (i.e. doing what lookups is it failing).

[Matthijsy]

Yes, when running this:

target = Target.open(Path("collect.zip"))
list(target.ad.ntds.users())

traceback:

Traceback (most recent call last):
  File "/Users/user/repos/dissect/dissect.target/test.py", line 22, in <module>
    list(target.ad.ntds.users())
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/ntds.py", line 94, in users
    yield from self.search(objectCategory="person", objectClass="user")
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/ntds.py", line 82, in search
    yield from self.db.data.search(**kwargs)
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/database.py", line 179, in search
    yield from self.query(f"(&{query})")
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/database.py", line 166, in query
    for record in Query(self.db, query, optimize=optimize).process():
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/query.py", line 36, in process
    yield from self._process_query(self._filter)
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/query.py", line 50, in _process_query
    yield from self._process_and_operation(filter, records)
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/query.py", line 113, in _process_and_operation
    for record in records_to_process:
                  ^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/query.py", line 56, in _process_query
    yield from self._query_database(filter)
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/ntds/query.py", line 92, in _query_database
    yield from index.cursor().find_all(**{schema.column: encoded_value})
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/cursor.py", line 191, in find_all
    record = self.record()
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/cursor.py", line 60, in record
    return Record(self.table, self._node())
                              ~~~~~~~~~~^^
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/cursor.py", line 51, in _node
    node = self._secondary.search(node.data.tobytes(), exact=True).node()
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/dissect313/lib/python3.13/site-packages/dissect/database/ese/cursor.py", line 381, in search
    raise KeyNotFoundError(f"Key not found: {key!r}")
dissect.database.ese.exception.KeyNotFoundError: Key not found: b'\x7f\x80\x00\x11y'