Skip to content

Commit ef12c72

Browse files
aramalloaramallo
committed
Merge branch 'release/0.3.2'
2 parents e6a40cd + e26cffa commit ef12c72

10 files changed

Lines changed: 191 additions & 59 deletions

File tree

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ examples/
2323
benchmark/cozodb_benchmark/_build/
2424
benchmark/cozodb_benchmark/deps/
2525
benchmark/cozodb_benchmark/benchmark_reports/
26+
benchmark/data/
27+
benchmark/reports/
2628

2729
# Ignore docs
2830
doc/

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
# CHANGELOG
2+
# 0.3.2
3+
## Bug Fixes
4+
* Fixed an issue when deploying in Docker. Jemalloc crashes when deploying in Docker Compose on Apple Silicon due to differences in OS page sizes - Added documentation to README
25

36
# 0.3.1
47

‎Makefile‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,16 @@ REBAR3 ?= $(shell test -e `which rebar3` 2>/dev/null && which rebar3 || echo "./
22

33
COZODB_TMP_DIR ?= "/tmp/cozodb/"
44

5+
# Jemalloc compile-time configuration for container compatibility
6+
# This bakes safe defaults directly into the binary - no runtime config needed
7+
# See: https://github.com/tikv/jemallocator/blob/master/jemalloc-sys/README.md
8+
#
9+
# background_thread:false - Prevents crashes in Docker/ECS/container environments
10+
# (background threads can fail after fork() in containers)
11+
# dirty_decay_ms:1000 - Balanced memory return to OS
12+
# muzzy_decay_ms:1000 - Balanced memory return to OS
13+
export JEMALLOC_SYS_WITH_MALLOC_CONF ?= background_thread:false,dirty_decay_ms:1000,muzzy_decay_ms:1000
14+
515
# RocksDB backend selection:
616
# COZODB_BACKEND=rocksdb (default) - Use cozorocks C++ FFI bridge
717
# COZODB_BACKEND=newrocksdb - Use rust-rocksdb crate with env var

‎README.md‎

Lines changed: 86 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,57 @@
1-
cozodb
2-
=====
1+
# cozodb
2+
3+
Erlang/BEAM NIF bindings for [CozoDB](https://github.com/cozodb/cozo) using Rustler.
4+
5+
CozoDB is a FOSS embeddable, transactional, relational-graph-vector database, with a Datalog query engine and time travelling capability, perfect as the long-term memory for LLMs and AI.
6+
7+
## Table of Contents
8+
9+
- [Quick Start](#quick-start)
10+
- [Installation](#installation)
11+
- [Requirements](#requirements)
12+
- [Erlang](#erlang)
13+
- [Elixir](#elixir)
14+
- [Basic Usage](#basic-usage)
15+
- [Storage Engines](#storage-engines)
16+
- [RocksDB Backends](#rocksdb-backends)
17+
- [rocksdb (cozorocks)](#1-rocksdb-cozorocks---default)
18+
- [newrocksdb (rust-rocksdb)](#2-newrocksdb-rust-rocksdb---alternative)
19+
- [Building with newrocksdb](#building-with-newrocksdb)
20+
- [Environment Variables](#newrocksdb-environment-variables)
21+
- [Configuration](#configuration)
22+
- [jemalloc](#jemalloc-configuration)
23+
- [Docker](#docker)
24+
- [Memory Tuning](#memory-tuning-tips)
25+
- [Development](#development)
26+
- [License](#license)
27+
28+
## Quick Start
329

4-
Erlang/BEAM NIF bindings for CozoDB using Rustler.
30+
```erlang
31+
%% Open an in-memory database
32+
{ok, Db} = cozodb:open(mem).
533

6-
CozoDB is a FOSS embeddable, transactional, relational-graph-vector database, wiht a Datalog query engine and time travelling capability, perfect as the long-term memory for LLMs and AI.
34+
%% Run a query
35+
{ok, Result} = cozodb:run(Db, "?[] <- [[1, 2, 3]]").
36+
37+
%% Close the database
38+
ok = cozodb:close(Db).
39+
```
740

841
## Installation
942

1043
### Requirements
11-
* Erlang OTP26 and/or Elixir (latest)
12-
* Rust 1.76.0
13-
* macOS packages:
14-
* `liblz4`,
15-
* `libssl`
16-
* Linux packages:
17-
* `build-essential`
18-
* `liblz4-dev`
19-
* `libncurses-dev`
20-
* `libsnappy-dev`
21-
* `libssl-dev`
22-
* `liburing-dev`
23-
* `liburing-dev`
24-
* `liburing2`
25-
* `pkg-config`
26-
27-
## Upgrading `cozo` dependency
28-
```bash
29-
cd native/cozodb
30-
cargo update -p cozo
31-
```
44+
45+
| Platform | Dependencies |
46+
|----------|--------------|
47+
| **Runtime** | Erlang OTP26+ and/or Elixir (latest) |
48+
| **Build** | Rust 1.76.0+ |
49+
| **macOS** | `liblz4`, `libssl` |
50+
| **Linux** | `build-essential`, `liblz4-dev`, `libncurses-dev`, `libsnappy-dev`, `libssl-dev`, `liburing-dev`, `liburing2`, `pkg-config` |
3251

3352
### Erlang
34-
Add the following to your `rebar.config` file.
53+
54+
Add the following to your `rebar.config` file:
3555

3656
```erlang
3757
{deps, [
@@ -42,44 +62,34 @@ Add the following to your `rebar.config` file.
4262
```
4363

4464
### Elixir
45-
Add the following to your `mix.exs` file.
65+
66+
Add the following to your `mix.exs` file:
4667

4768
```elixir
48-
defp deps do
49-
[
50-
{:cozodb,
51-
git: "https://github.com/leapsight/cozodb.git",
52-
branch: "master"
53-
}
54-
]
69+
defp deps do
70+
[
71+
{:cozodb,
72+
git: "https://github.com/leapsight/cozodb.git",
73+
branch: "master"
74+
}
75+
]
76+
end
5577
```
5678

57-
### Using newrocksdb Backend as a Consumer
58-
59-
By default, cozodb compiles with the `rocksdb` (cozorocks) backend. If you need the `newrocksdb` backend with environment variable configuration, you have two options:
60-
61-
#### Option A: Fork and Modify (Recommended)
79+
## Basic Usage
6280

63-
1. Fork the cozodb repository
64-
2. Modify `native/cozodb/Cargo.toml` to use `new-rocksdb-default` as the default features
65-
3. Point your dependency to your fork
66-
67-
#### Option B: Pre-build the NIF
68-
69-
Before running `rebar3 compile` in your project:
81+
```erlang
82+
%% In-memory database
83+
{ok, Db} = cozodb:open(mem).
7084

71-
```bash
72-
# Clone cozodb
73-
git clone https://github.com/leapsight/cozodb.git /tmp/cozodb
74-
cd /tmp/cozodb
85+
%% SQLite database
86+
{ok, Db} = cozodb:open(sqlite, "/path/to/db.sqlite").
7587

76-
# Build with newrocksdb backend
77-
COZODB_BACKEND=newrocksdb make cargo-build
88+
%% RocksDB database (default backend)
89+
{ok, Db} = cozodb:open(rocksdb, "/path/to/db").
7890

79-
# Copy the built NIF to your project's _build directory
80-
# (after rebar3 has fetched dependencies)
81-
mkdir -p YOUR_PROJECT/_build/default/lib/cozodb/priv/crates/cozodb
82-
cp priv/crates/cozodb/cozodb.so YOUR_PROJECT/_build/default/lib/cozodb/priv/crates/cozodb/
91+
%% New RocksDB backend (if compiled with new-rocksdb-default feature)
92+
{ok, Db} = cozodb:open(newrocksdb, "/path/to/db").
8393
```
8494

8595
## Storage Engines
@@ -282,6 +292,26 @@ The NIF uses jemalloc for memory management. Configure via environment variables
282292
| `COZODB_JEMALLOC_BACKGROUND_THREAD` | bool | true | Enable background purging thread |
283293
| `COZODB_JEMALLOC_NARENAS` | u32 | auto | Number of arenas (optional) |
284294

295+
### Docker
296+
#### Build the Rust NIF with C++20 support (required by newer RocksDB headers)
297+
```Dockerfile
298+
ENV CXXFLAGS="-std=c++20"
299+
```
300+
#### CRITICAL jemalloc Config
301+
* Set page size for jemalloc to match Linux x86_64 (4KB = 2^12) -- This prevents "page size mismatch" errors when running in Docker containers. Different from macOS Apple Silicon which uses 16KB (LG_PAGE=14) or 64KB pages.
302+
303+
Add the following to your Dockerfile:
304+
305+
```Dockerfile
306+
ENV JEMALLOC_SYS_WITH_LG_PAGE=12
307+
```
308+
309+
Also for container compatibility set `JEMALLOC_SYS_WITH_MALLOC_CONF`. This bakes safe defaults directly into the binary - no runtime MALLOC_CONF needed
310+
See: https://github.com/tikv/jemallocator/blob/master/jemalloc-sys/README.md
311+
312+
```Dockerfile
313+
ENV JEMALLOC_SYS_WITH_MALLOC_CONF="background_thread:false,dirty_decay_ms:1000,muzzy_decay_ms:1000"
314+
```
285315
### Memory Tuning Tips
286316

287317
- **Low memory usage:** Set decay values to 0 for aggressive memory return

‎benchmark/Dockerfile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,18 @@ COPY . .
2828

2929
# Build the Rust NIF with C++20 support (required by newer RocksDB headers)
3030
ENV CXXFLAGS="-std=c++20"
31+
32+
# CRITICAL: Compile-time jemalloc configuration for container compatibility
33+
# This bakes safe defaults directly into the binary - no runtime MALLOC_CONF needed
34+
# See: https://github.com/tikv/jemallocator/blob/master/jemalloc-sys/README.md
35+
ENV JEMALLOC_SYS_WITH_MALLOC_CONF="background_thread:false,dirty_decay_ms:1000,muzzy_decay_ms:1000"
36+
37+
# CRITICAL: Set page size for jemalloc to match Linux x86_64 (4KB = 2^12)
38+
# This prevents "page size mismatch" errors when running in Docker containers
39+
# Different from macOS Apple Silicon which uses 16KB (LG_PAGE=14) or 64KB pages
40+
ENV JEMALLOC_SYS_WITH_LG_PAGE=12
41+
42+
3143
RUN cd native/cozodb && cargo build --release
3244

3345
# Copy the built NIF to priv directory

‎benchmark/docker-compose.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Docker Compose file for CozoDB benchmark
2+
# Compatible with Docker Compose and AWS ECS
3+
#
4+
# The jemalloc configuration is now baked into the binary at compile time
5+
# via JEMALLOC_SYS_WITH_MALLOC_CONF in the Dockerfile. No runtime config needed.
6+
#
7+
# Usage:
8+
# docker-compose up --build # Build and run
9+
# docker-compose up # Run (after build)
10+
# docker-compose down # Stop and remove
11+
12+
services:
13+
cozodb-benchmark:
14+
build:
15+
context: ..
16+
dockerfile: benchmark/Dockerfile
17+
18+
# Use init for proper signal handling and process reaping
19+
# In ECS task definition: "initProcessEnabled": true
20+
init: true
21+
22+
# Resource limits (compatible with ECS task definitions)
23+
deploy:
24+
resources:
25+
limits:
26+
cpus: '2'
27+
memory: 2G
28+
reservations:
29+
cpus: '1'
30+
memory: 1G
31+
32+
environment:
33+
# Optional: Override the compile-time defaults if needed
34+
# The defaults (background_thread:false, decay_ms:1000) are baked into the binary
35+
- COZODB_JEMALLOC_DIRTY_DECAY_MS=1000
36+
- COZODB_JEMALLOC_MUZZY_DECAY_MS=1000
37+
38+
volumes:
39+
- benchmark-data:/tmp/cozodb_benchmark
40+
- benchmark-reports:/app/benchmark_reports
41+
42+
healthcheck:
43+
test: ["CMD", "pgrep", "-f", "beam"]
44+
interval: 30s
45+
timeout: 10s
46+
retries: 3
47+
start_period: 60s
48+
49+
logging:
50+
driver: json-file
51+
options:
52+
max-size: "10m"
53+
max-file: "3"
54+
55+
volumes:
56+
benchmark-data:
57+
benchmark-reports:

‎native/cozodb/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
name = "erlang-cozodb"
33
description = "Erlang NIF wrapper for CozoDB using Rustler."
44
authors = ["Alejandro M. Ramallo"]
5-
version = "0.3.1"
5+
version = "0.3.2"
66
edition = "2021"
77

88
[lib]

‎native/cozodb/src/lib.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,24 @@ use tikv_jemallocator::Jemalloc;
5151
#[global_allocator]
5252
static GLOBAL: Jemalloc = Jemalloc;
5353

54+
// =============================================================================
55+
// JEMALLOC COMPILE-TIME CONFIGURATION
56+
// =============================================================================
57+
// This static variable is read by jemalloc BEFORE main() is entered.
58+
// It provides safe defaults for containerized environments (Docker, ECS, K8s).
59+
//
60+
// Key settings:
61+
// - background_thread:false - Prevents crashes after fork() in containers
62+
// - dirty_decay_ms:1000 - Balanced memory return to OS (1 second)
63+
// - muzzy_decay_ms:1000 - Balanced memory return to OS (1 second)
64+
//
65+
// These can be overridden at runtime via MALLOC_CONF environment variable.
66+
// =============================================================================
67+
#[cfg(all(feature = "jemalloc", not(feature = "nif_alloc"), not(target_env = "msvc")))]
68+
#[allow(non_upper_case_globals)]
69+
#[export_name = "malloc_conf"]
70+
pub static malloc_conf: &[u8] = b"background_thread:false,dirty_decay_ms:1000,muzzy_decay_ms:1000\0";
71+
5472
// Rust std libs
5573

5674
use core::hash::Hash;

‎rebar.config‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
observer_cli
2525
]},
2626
{relx, [
27-
{release, {cozodb, "0.3.1"}, [
27+
{release, {cozodb, "0.3.2"}, [
2828
%% Erlang
2929
sasl,
3030
crypto,

‎src/cozodb.app.src‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{application, cozodb, [
22
{description, "Erlang NIF wrapper for CozoDB using Rustler."},
3-
{vsn, "0.3.1"},
3+
{vsn, "0.3.2"},
44
{registered, []},
55
{mod, {cozodb_app, []}},
66
{applications, [

0 commit comments

Comments
 (0)