Skip to content

feat: export a compact version of the database #1740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
marcoconti83 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: export a compact version of the database #1740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,8 @@ $(TS_OUT): $(ts-deps)
--target browser \
--format esm \
--outdir out \
--entry-naming "corecrypto.[ext]" && \
--entry-naming "corecrypto.[ext]" \
--external env && \
mkdir -p out/autogenerated/ && \
mkdir -p out/autogenerated/wasm-bindgen && \
cp src/autogenerated/wasm-bindgen/index_bg.wasm out/autogenerated/wasm-bindgen/ && \
Expand Down
1 change: 1 addition & 0 deletions crypto-ffi/bindings/js/src/CoreCrypto.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ export {
type MlsTransportData,
migrateDatabaseKeyTypeToBytes,
updateDatabaseKey,
exportDatabaseCopy,
NewAcmeAuthz,
NewAcmeOrder,
ExternalSenderKey,
Expand Down
39 changes: 39 additions & 0 deletions crypto-ffi/src/database/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,42 @@ pub async fn open_database(name: &str, key: Arc<DatabaseKey>) -> CoreCryptoResul
pub async fn in_memory_database(key: Arc<DatabaseKey>) -> CoreCryptoResult<Database> {
Database::in_memory(key).await
}

/// Export a copy of the database to the specified path.
///
/// This creates a fully vacuumed and optimized copy of the database using SQLite's VACUUM INTO command.
/// The copy will be encrypted with the same key as the source database.
///
/// # Platform Support
/// This method is only available on platforms using SQLCipher (iOS, Android, JVM, native).
/// On WASM platforms, this function will return an error.
///
/// # Arguments
/// * `database` - The database instance to export
/// * `destination_path` - The file path where the database copy should be created
///
/// # Errors
/// Returns an error if:
/// - Called on WASM platform (not supported)
/// - The database is in-memory (cannot export in-memory databases)
/// - The destination path is invalid or not writable
/// - The export operation fails
#[uniffi::export]
pub async fn export_database_copy(database: &Database, destination_path: &str) -> CoreCryptoResult<()> {
#[cfg(target_family = "wasm")]
{
let _ = (database, destination_path); // Suppress unused warnings
Err(CoreCryptoError::ad_hoc(
"export_database_copy is not supported on WASM. This function requires filesystem operations and SQLCipher, which are only available on native platforms (iOS, Android, JVM).",
))
}

#[cfg(not(target_family = "wasm"))]
{
database
.0
.export_copy(destination_path)
.await
.map_err(CoreCryptoError::generic())
}
}
Comment on lines +65 to +82
Copy link
Member

@SimonThormeyer SimonThormeyer Jan 13, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we feature-gated this entire function instead:

#[cfg(not(feature = "wasm"))]

we could prevent this function to be generated for typescript entirely. Or was there a particular reason why you wanted to expose this function for TypeScript?

Copy link
Member Author

@marcoconti83 marcoconti83 Jan 14, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to feature-gate the entire function. I then spent hours trying to get the WASM tests to run - they would fail, because even if the tests were not using this function, there was a step in the test setup that checks that all functions part of the FFI are defined, and it would fail. I didn't dig deeper into why this happens, but removing this check seems like a big architectural change that is beyond the scope of this PR.

Auto-generating stubs for WASM did not work because the function just does not exists when using the WASM target. So I tried manually generating stubs, but either we hardcode the stubs (and we need to keep them in sync with the code as the code changes), or we would have a script that looks for functions that don't exists and generates stubs for them (which feels very hacky).
I decided to just throw an error when running this in WASM, and this makes the tests happy.

Copy link
Member Author

@marcoconti83 marcoconti83 Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this function will only ever be called by Android anyway

Copy link
Member

@SimonThormeyer SimonThormeyer Jan 14, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure that you annotated the function with exactly the attribute I'm mentioning above?

It's unintuitive at first that if we want to hide a part of the API for typescript, we cannot compile conditionnally by target family, but have to use that feature-gate. It's due to how uniffi-bindgen-for-react-native (ubrn) generates the bindings. They are generated from a binary that is compiled for the host platform. So if you'd compile conditionally by target family, the function would still appear in the binary the bindings are generated from, which would then try to bind to a function that doesn't exist on the target.

I did an experiment in 96307cc where I feature-gate the function. Typescript tests are passing there for me.

Copy link
Contributor

@coriolinus coriolinus Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO building on Simon's work is the way to go here. Possibly that will also solve the interop test issue?

Copy link
Member

@SimonThormeyer SimonThormeyer Jan 14, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interop test issue will probably be fixed with a rebase on main: on main, we're creating android emulators freshly for each android test (including interop), and this branch assumes emulators are reused across test runs.

3 changes: 2 additions & 1 deletion crypto-ffi/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ pub use credential_ref::CredentialRef;
pub use credential_type::CredentialType;
pub use crl::CrlRegistration;
pub use database::{
Database, DatabaseKey, in_memory_database, migrate_database_key_type_to_bytes, open_database, update_database_key,
Database, DatabaseKey, export_database_copy, in_memory_database, migrate_database_key_type_to_bytes, open_database,
update_database_key,
};
pub use decrypted_message::{BufferedDecryptedMessage, DecryptedMessage};
pub use e2ei::{
Expand Down
21 changes: 21 additions & 0 deletions keystore/src/connection/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,27 @@ impl Database {
self.take().await?.wipe().await
}

/// Export a copy of the database to the specified path.
/// This creates a fully vacuumed and optimized copy of the database.
/// The copy will be encrypted with the same key as the source database.
///
/// # Platform Support
/// This method is only available on platforms using SQLCipher (not WASM).
///
/// # Arguments
/// * `destination_path` - The file path where the database copy should be created
///
/// # Errors
/// Returns an error if:
/// - The database is in-memory (cannot export in-memory databases)
/// - The destination path is invalid
/// - The export operation fails
#[cfg(not(target_family = "wasm"))]
pub async fn export_copy(&self, destination_path: &str) -> CryptoKeystoreResult<()> {
let conn = self.conn().await?;
conn.export_copy(destination_path).await
}

pub async fn migrate_db_key_type_to_bytes(
name: &str,
old_key: &str,
Expand Down
114 changes: 112 additions & 2 deletions keystore/src/connection/platform/generic/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,26 @@ impl SqlCipherConnection {
Ok(())
}

/// Export a copy of the database to the specified path using VACUUM INTO.
///
/// This creates a fully vacuumed and optimized copy of the database.
/// The copy will be encrypted with the same key as the source database.
///
/// # Arguments
/// * `destination_path` - The file path where the database copy should be created
///
/// # Errors
/// Returns an error if:
/// - The database is in-memory (cannot export in-memory databases)
/// - The destination path is invalid
/// - The VACUUM INTO operation fails
pub async fn export_copy(&self, destination_path: &str) -> CryptoKeystoreResult<()> {
let conn = self.conn().await;
conn.execute("VACUUM INTO ?1", [destination_path])?;

Ok(())
}

fn run_migrations(conn: &mut rusqlite::Connection, target: MigrationTarget) -> CryptoKeystoreResult<()> {
conn.create_scalar_function("sha256_blob", 1, FunctionFlags::SQLITE_DETERMINISTIC, |ctx| {
let input_blob = ctx.get::<Vec<u8>>(0)?;
Expand Down Expand Up @@ -327,8 +347,11 @@ mod migration_test {
entities::{EntityFindParams, StoredCredential},
};

const DB: &[u8] = include_bytes!("../../../../../crypto-ffi/bindings/jvm/src/test/resources/db-v10002003.sqlite");
const OLD_KEY: &str = "secret";
pub(super) const DB: &[u8] = include_bytes!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../crypto-ffi/bindings/jvm/src/test/resources/db-v10002003.sqlite"
));
pub(super) const OLD_KEY: &str = "secret";

// a close replica of the JVM test in `GeneralTest.kt`, but way more debuggable
#[test]
Expand Down Expand Up @@ -456,3 +479,90 @@ mod migration_test {
});
}
}

#[cfg(test)]
mod export_test {
use futures_lite::future;

use crate::{ConnectionType, Database, DatabaseKey};

#[test]
fn can_export_database_copy() {
future::block_on(async {
// Create temporary directory
let temp_dir = tempfile::tempdir().unwrap();
let source_path = temp_dir.path().join("test_export_source.db");
let dest_path = temp_dir.path().join("test_export_dest.db");

// Write test database
std::fs::write(&source_path, super::migration_test::DB).unwrap();

// Migrate the database to use the new key format
let key = DatabaseKey::generate();
Database::migrate_db_key_type_to_bytes(source_path.to_str().unwrap(), super::migration_test::OLD_KEY, &key)
.await
.unwrap();

// Open the database
let db = Database::open(ConnectionType::Persistent(source_path.to_str().unwrap()), &key)
.await
.unwrap();

// Insert test data into a test table
let test_data = b"test data for export verification";
let test_id = 12345;
{
let conn = db.conn().await.unwrap();
let conn_guard = conn.conn().await;

// Create a test table
conn_guard
.execute(
"CREATE TABLE IF NOT EXISTS test_export_data (id INTEGER PRIMARY KEY, data BLOB)",
[],
)
.unwrap();

// Insert test data
conn_guard
.execute(
"INSERT INTO test_export_data (id, data) VALUES (?1, ?2)",
[&test_id as &dyn rusqlite::ToSql, &test_data.as_slice()],
)
.unwrap();
}

// Export the database
db.export_copy(dest_path.to_str().unwrap()).await.unwrap();

// Verify the exported database can be opened with the same key
let exported_db = Database::open(ConnectionType::Persistent(dest_path.to_str().unwrap()), &key)
.await
.unwrap();

// Read the data from the exported database
{
let conn = exported_db.conn().await.unwrap();
let conn_guard = conn.conn().await;

let mut stmt = conn_guard
.prepare("SELECT id, data FROM test_export_data WHERE id = ?1")
.unwrap();
let mut rows = stmt.query([test_id]).unwrap();

let row = rows.next().unwrap().expect("Expected row to exist");
let read_id: i32 = row.get(0).unwrap();
let read_data: Vec<u8> = row.get(1).unwrap();

assert_eq!(read_id, test_id, "ID should match in exported database");
assert_eq!(read_data, test_data, "Data should match in exported database");
}

// Close databases before cleanup
drop(db);
drop(exported_db);

// temp_dir is automatically cleaned up when it goes out of scope
});
}
}
2 changes: 2 additions & 0 deletions keystore/src/error.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ pub enum CryptoKeystoreError {
JsError(String),
#[error("Not implemented (and probably never will)")]
NotImplemented,
#[error("Operation not supported: {0}")]
NotSupported(String),
#[error("Failed getting current timestamp")]
TimestampError,
#[error("Could not find {0} in keystore with value {1}")]
Expand Down
Loading