Improve SQLite3 WAL by PimSanders · Pull Request #9 · fox-it/dissect.database

PimSanders · 2025-11-19T15:19:37Z

This PR:

Closes Implement opening DB's from various WAL checkpoints #12 by implementing a feature to open databases from various WAL checkpoints.
Improves the reader by also parsing the WAL file
Allows a database to be initialized with a Path OR BinaryIO (as sugested in Add Windows Search Index plugin dissect.target#1254).
Adds a Python script to generate test data (as sugested in Add scripts to generate reproducible test data dissect#95 ).

A WAL file and/or WAL checkpoint can be opened on initialization:

sqlite3.SQLite3(sqlite_db, sqlite_wal, wal_checkpoint=2)

Previous WAL checkpoints might contain interesting data that has been modified or removed, for example:

Checkpoints found: 4
Salt1: 1945264100
[<Row table=test id=1 name='testing' value=1337>,
 <Row table=test id=2 name='omg' value=7331>,
 <Row table=test id=3 name='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' value=4100>,
 <Row table=test id=4 name='BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' value=4100>,
 <Row table=test id=5 name='negative' value=-11644473429>,
 <Row table=test id=6 name='after checkpoint' value=42>,
 <Row table=test id=8 name='after checkpoint' value=44>,
 <Row table=test id=9 name='wow' value=1234>,
 <Row table=test id=10 name='second checkpoint' value=100>,
 <Row table=test id=11 name='second checkpoint' value=101>]
Salt1: 1945264099
[<Row table=test id=1 name='testing' value=1337>,
 <Row table=test id=2 name='omg' value=7331>,
 <Row table=test id=3 name='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' value=4100>,
 <Row table=test id=4 name='BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' value=4100>,
 <Row table=test id=5 name='negative' value=-11644473429>,
 <Row table=test id=6 name='after checkpoint' value=42>,
 <Row table=test id=7 name='after checkpoint' value=43>,
 <Row table=test id=8 name='after checkpoint' value=44>,
 <Row table=test id=9 name='after checkpoint' value=45>,
 <Row table=test id=10 name='second checkpoint' value=100>,
 <Row table=test id=11 name='second checkpoint' value=101>]
Salt1: 1945264098
[<Row table=test id=1 name='testing' value=1337>,
 <Row table=test id=2 name='omg' value=7331>,
 <Row table=test id=3 name='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' value=4100>,
 <Row table=test id=4 name='BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' value=4100>,
 <Row table=test id=5 name='negative' value=-11644473429>,
 <Row table=test id=6 name='after checkpoint' value=42>,
 <Row table=test id=7 name='after checkpoint' value=43>,
 <Row table=test id=8 name='after checkpoint' value=44>,
 <Row table=test id=9 name='after checkpoint' value=45>]
Salt1: 1945264097
[<Row table=test id=1 name='testing' value=1337>,
 <Row table=test id=2 name='omg' value=7331>,
 <Row table=test id=3 name='AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' value=4100>,
 <Row table=test id=4 name='BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' value=4100>,
 <Row table=test id=5 name='negative' value=-11644473429>]

If no checkpoint is specified the reader will use the latest valid data from the WAL file, if that's not available it will use the database file. If no WAL file is provided it will check if DB.sqlite-wal or DB.db-wal exists.

There are three things I'm not sure about and I would like your opinion on:

Currently checkpoints are ordered in reverse, this means that checkpoint 0 is the latest and checkpoint 3 is the oldest. Is this the right call or should they be inverted and follow the checkpoint_sequence counter in the header?
The Python script to generate is currently stored in tests/_data which means it will be saved in LFS, I don't think it should be stored in LFS. Should the script stay in _data and have an exception from LFS or should it be stored somewhere else?
The test could be shorter, but I think it is best to test for everything every time, right?

This commit: - Adds a feature to open databases from various WAL checkpoints. - Allows a database to be initialized with a Path OR BinaryIO. - Improves the reader by also parsing the WAL file - Adds a Python script to generate test data.

codecov · 2025-11-19T15:21:00Z

Codecov Report

❌ Patch coverage is 86.30137% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (14741f8) to head (e12b54b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dissect/database/sqlite3/wal.py	86.60%	15 Missing ⚠️
dissect/database/sqlite3/sqlite3.py	85.29%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   79.71%   81.91%   +2.20%     
==========================================
  Files          30       31       +1     
  Lines        2282     2334      +52     
==========================================
+ Hits         1819     1912      +93     
+ Misses        463      422      -41

Flag	Coverage Δ
unittests	`81.91% <86.30%> (+2.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Schamper

Can you move all WAL related classes to a file wal.py and lose the WAL prefix in the class names?