From b46a21ea307965abd63cd52e9c45f46e1845efb3 Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:31:04 +0100 Subject: [PATCH 1/8] Add documentation for running and organizing tests Added TESTS.md to describe test directory structure, local and CI test execution, and best practices. Added integration/README.md with detailed instructions for running integration tests, environment setup, and CI recommendations. --- test/TESTS.md | 62 ++++++++++++++++++++++++ test/integration/README.md | 98 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 test/TESTS.md create mode 100644 test/integration/README.md diff --git a/test/TESTS.md b/test/TESTS.md new file mode 100644 index 0000000..74b9060 --- /dev/null +++ b/test/TESTS.md @@ -0,0 +1,62 @@ +# Tests — Quick reference + +This document explains how tests are organised in the repository and how to run them locally and in CI. + +Directory layout + +- `test/sparkling/` — unit and component tests that do not require external services (fast). +- `test/smoke/` — smoke/sanity tests. Quick checks that the test runner and a minimal API behave as expected. +- `test/integration/` — integration tests that require external services (e.g. ClickHouse). These are slower and should not run on every PR. + +Running tests locally + +- Run the full test suite (if you have required services available): + +```bash +# from the project root +gleam test +``` + +- Run unit tests only (recommended for PRs): + +```bash +# if your test runner accepts paths: +gleam test test/sparkling +``` + +If your runner does not accept paths, run `gleam test` locally and ensure integration tests are not executed by default (integration tests should be placed under `test/integration/`). + +- Run integration tests (requires ClickHouse or other external services): + +```bash +# Start ClickHouse (example using Docker) +docker run -d --name clickhouse-server -p 8123:8123 -p 9000:9000 clickhouse/clickhouse-server:latest + +# Run integration tests +gleam test test/integration + +# When finished, stop/remove the container +docker stop clickhouse-server && docker rm clickhouse-server +``` + +Environment variables + +- Use environment variables to configure integration endpoints (example): + +```bash +export CLICKHOUSE_URL=http://localhost:8123 +``` + +Document required variables in `test/integration/README.md`. + +CI guidance + +- Pull Requests: run unit tests only. +- Integration tests: run in separate CI jobs (manual trigger, nightly, or on release tags). Ensure the CI environment has access to required services and secrets before enabling integration jobs. + +Best practices and PR checklist + +- [ ] Unit tests pass locally (`gleam test`). +- [ ] Integration tests are documented in `test/integration/README.md` with clear prerequisites and run instructions. +- [ ] Do not add CI workflows that perform sensitive operations (publishing) in the import PR. Add automation in a separate PR after the code is stable. +- [ ] Keep a lightweight smoke test under `test/smoke/` to validate the test runner and minimal API surface. \ No newline at end of file diff --git a/test/integration/README.md b/test/integration/README.md new file mode 100644 index 0000000..e4cca4e --- /dev/null +++ b/test/integration/README.md @@ -0,0 +1,98 @@ +# Integration tests + +Integration tests that exercise the code against a real ClickHouse instance. + +This document explains how to run integration tests locally and how to wire them into CI in a safe way. + +Prerequisites + +- Docker (used for the example below). +- docker-compose (optional, if you prefer a compose-based setup). + +Recommended local setup (Docker Compose) + +1. Start ClickHouse with docker-compose (example): + +```bash +# from the repository root, if you have a docker-compose.yml configured +docker-compose up -d +``` + +2. Wait for the server to be ready: + +```bash +docker-compose ps +``` + +3. Run only the integration tests: + +```bash +# from the repository root +gleam test test/integration +``` + +4. Tear down the environment when finished: + +```bash +docker-compose down -v +``` + +Quick single-container alternative (no compose) + +```bash +docker run -d --name clickhouse-server -p 8123:8123 -p 9000:9000 clickhouse/clickhouse-server:latest +# wait for readiness, then run tests +gleam test test/integration +docker stop clickhouse-server && docker rm clickhouse-server +``` + +Connection details (local defaults) + +- URL: http://localhost:8123 +- Database: test_db (tests may create/drop their own DBs) +- User: test_user +- Password: test_password + +If your tests require different credentials or endpoints, set the appropriate environment variables (see the `Environment variables` section below). + +Implemented integration tests + +- `basic_connectivity_test.gleam` — verifies connection and basic queries +- `complex_types_roundtrip_test.gleam` — round-trip tests for complex ClickHouse types +- `format_compatibility_test.gleam` — tests different I/O formats (JSONEachRow, CSV, TabSeparated) +- `performance_test.gleam` — optional performance/load checks (may be slow) + +Environment variables + +Use environment variables to configure endpoints and credentials for integration tests. Example: + +```bash +export CLICKHOUSE_URL=http://localhost:8123 +export CLICKHOUSE_USER=test_user +export CLICKHOUSE_PASSWORD=test_password +``` + +Document required environment variables and secrets clearly before enabling integration jobs in CI. + +CI guidance + +- Integration tests should not run by default on every PR. Run them in separate CI jobs, e.g. on manual dispatch, nightly builds, or release tags. +- Example GitHub Actions job (run in a dedicated workflow or as a separate job): + +```yaml +- name: Integration tests (optional) + run: | + docker-compose up -d + # optionally wait / healthcheck + sleep 10 + gleam test test/integration + docker-compose down -v + # do not fail the whole pipeline if integration env is missing + continue-on-error: true +``` + +Notes and recommendations + +- Keep integration tests isolated and idempotent: tests should create and drop any resources they need. +- Mark expensive or flaky tests (e.g. `performance_test.gleam`) so CI can skip them unless explicitly requested. +- Before enabling integration tests in CI, ensure any required secrets or credentials are set in the CI environment. From 7b461c4ad87c3cfe2088457d2a849c7ec13902ec Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:31:28 +0100 Subject: [PATCH 2/8] Add CI, integration, and smoke test workflows Introduces GitHub Actions workflows for continuous integration, integration testing across multiple ClickHouse versions, and smoke integration tests on pull requests. These workflows automate building, formatting, unit testing, and basic integration checks to ensure code quality and compatibility. --- workflows/ci.yml | 48 +++++++++++++++++++ workflows/integration.yml | 81 +++++++++++++++++++++++++++++++++ workflows/smoke-integration.yml | 65 ++++++++++++++++++++++++++ 3 files changed, 194 insertions(+) create mode 100644 workflows/ci.yml create mode 100644 workflows/integration.yml create mode 100644 workflows/smoke-integration.yml diff --git a/workflows/ci.yml b/workflows/ci.yml new file mode 100644 index 0000000..cf0dbb3 --- /dev/null +++ b/workflows/ci.yml @@ -0,0 +1,48 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + unit-tests: + name: Unit Tests + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Erlang/OTP and Gleam + uses: erlef/setup-beam@v1 + with: + otp-version: "26.0" + gleam-version: "1.11.0" + + - name: Show Gleam version + run: gleam --version + + - name: Cache Gleam deps and build + uses: actions/cache@v4 + with: + path: | + ~/.cache/gleam + ~/.gleam + ./_gleam_deps + ./build + key: ${{ runner.os }}-gleam-1.11.0-cache-v1-${{ hashFiles('**/gleam.toml') }} + restore-keys: | + ${{ runner.os }}-gleam-1.11.0-cache-v1- + + - name: Install dependencies + run: gleam deps download + + - name: Check formatting + run: gleam format --check src test + + - name: Build project + run: gleam build + + - name: Run unit tests + run: gleam test diff --git a/workflows/integration.yml b/workflows/integration.yml new file mode 100644 index 0000000..8f83818 --- /dev/null +++ b/workflows/integration.yml @@ -0,0 +1,81 @@ +name: Integration tests + +on: + workflow_dispatch: + push: + branches: ["main"] + +jobs: + integration-tests: + runs-on: ubuntu-latest + concurrency: + group: integration-tests-${{ matrix.clickhouse_version }} + cancel-in-progress: false + timeout-minutes: 90 + strategy: + matrix: + clickhouse_version: ["22.8", "23.7", "24.4", "latest"] + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Cache Gleam build artifacts + uses: actions/cache@v4 + with: + path: | + ./build + ~/.cache/gleam + key: ${{ runner.os }}-gleam-1.11.0-integration-${{ matrix.clickhouse_version }}-v1-${{ hashFiles('**/gleam.toml') }} + + - name: Start ClickHouse ${{ matrix.clickhouse_version }} (docker-compose) + env: + CLICKHOUSE_IMAGE: clickhouse/clickhouse-server:${{ matrix.clickhouse_version }} + run: | + echo "Using CLICKHOUSE_IMAGE=$CLICKHOUSE_IMAGE" + # Ensure docker compose is available (Docker-hosted runners include docker) + docker compose pull clickhouse || true + docker compose up -d --remove-orphans clickhouse + + - name: Wait for ClickHouse HTTP + run: | + echo "Waiting for ClickHouse HTTP on localhost:8123..." + for i in $(seq 1 60); do + if curl -sSf http://localhost:8123/ >/dev/null 2>&1; then + echo "ClickHouse is up" + break + fi + sleep 2 + done + + - name: Setup Erlang/OTP and Gleam + uses: erlef/setup-beam@v1 + with: + otp-version: "26.0" + gleam-version: "1.11.0" + + - name: Show Gleam version + run: gleam --version + + - name: Run integration tests + env: + CLICKHOUSE_USER: test_user + CLICKHOUSE_PASSWORD: test_password + CLICKHOUSE_DB: test_db + CLICKHOUSE_URL: http://localhost:8123 + run: | + gleam build + gleam test + + - name: Upload integration test artifacts (logs) + if: failure() + uses: actions/upload-artifact@v4 + with: + name: integration-logs-${{ matrix.clickhouse_version }} + path: | + ./build/logs || true + + - name: Cleanup ClickHouse + if: always() + run: | + docker compose down -v || true diff --git a/workflows/smoke-integration.yml b/workflows/smoke-integration.yml new file mode 100644 index 0000000..5881ff0 --- /dev/null +++ b/workflows/smoke-integration.yml @@ -0,0 +1,65 @@ +name: Smoke Integration (PR) + +on: + pull_request: + branches: ["main"] + +jobs: + smoke-integration: + name: Smoke Integration (ClickHouse) + runs-on: ubuntu-latest + timeout-minutes: 30 + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Start ClickHouse (docker-compose) + env: + CLICKHOUSE_IMAGE: clickhouse/clickhouse-server:23.7 + run: | + echo "Using CLICKHOUSE_IMAGE=$CLICKHOUSE_IMAGE" + docker compose pull clickhouse || true + docker compose up -d --remove-orphans clickhouse + + - name: Wait for ClickHouse HTTP + run: | + echo "Waiting for ClickHouse HTTP on localhost:8123..." + for i in $(seq 1 40); do + if curl -sSf http://localhost:8123/ >/dev/null 2>&1; then + echo "ClickHouse is up" + break + fi + sleep 2 + done + + - name: Smoke test - SELECT 1 + run: | + set -e + OUT=$(curl -sS -u test_user:test_password "http://localhost:8123/?query=SELECT%201%20as%20result%20FORMAT%20JSONEachRow&database=test_db") + echo "Got: $OUT" + if [ "$OUT" != '{"result":1}' ] && [ "$OUT" != '{"result":1}\n' ]; then + echo "Unexpected response: $OUT" + exit 2 + fi + + - name: Smoke test - create table, insert and select + run: | + set -e + DDL="CREATE TABLE IF NOT EXISTS smoke_test (id UInt32, name String) ENGINE = MergeTree() ORDER BY id" + curl -sS -u test_user:test_password -d "$DDL" "http://localhost:8123/?database=test_db" + + INSERT="INSERT INTO smoke_test (id,name) VALUES (1,'smoke')" + curl -sS -u test_user:test_password -d "$INSERT" "http://localhost:8123/?database=test_db" + + OUT=$(curl -sS -u test_user:test_password "http://localhost:8123/?query=SELECT%20*%20FROM%20smoke_test%20WHERE%20id%3D1%20FORMAT%20JSONEachRow&database=test_db") + echo "Select returned: $OUT" + if [ -z "$OUT" ]; then + echo "Select returned empty" + exit 3 + fi + + - name: Cleanup ClickHouse + if: always() + run: | + docker compose down -v || true From d5b83195fb7aa3e898ccf14f80de09b0a7db614f Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:31:34 +0100 Subject: [PATCH 3/8] Update README with CI badges and quick start guide Added CI and integration test badges, a Gleam version badge, and a new Quick Start section with example usage. Expanded documentation to clarify available modules and design inspiration. --- README.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ed6d532..913bdf6 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Sparkling logo

-[![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) +[![CI](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml) [![Integration Tests](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) [![Gleam Version](https://img.shields.io/badge/gleam-%3E%3D1.11.0-ffaff3)](https://gleam.run) **Sparkling** is a *lightweight*, **type-safe** data layer for **ClickHouse** written in Gleam. It provides a small, focused API for defining schemas, building queries, and encoding/decoding ClickHouse formats. @@ -12,6 +12,37 @@ > Why "Sparkling"? One rainy Tuesday a tiny inflatable rubber duck stole a shooting pink star and decided to become a freelance data wrangler, and it now guides queries through the night, humming 8-bit lullabies. Totally plausible. +## Quick start + +See the extracted quick start example: `docs/quickstart.md` it contains a short walkthrough (define schema, build a query, execute it with a repo). + +Minimal example: + +```gleam +import sparkling/repo + +let r = repo.new("http://localhost:8123") + |> repo.with_database("mydb") + +case r.execute_sql(r, "SELECT 1 as result FORMAT JSONEachRow") { + Ok(body) -> io.println(body) + Error(_) -> io.println("query failed") +} +``` + +## What you'll find here + +- `sparkling/schema` — typed table & column definitions +- `sparkling/query` — immutable query builder (to_sql) +- `sparkling/repo` — HTTP executor with retry hooks +- `sparkling/encode` / `sparkling/decode` — format handlers (JSONEachRow default) +- `sparkling/types` — helpers for Decimal, DateTime64, UUID, LowCardinality + +For more examples see `docs/examples/` and `docs/quickstart.md`. + +**Design note:** Sparkling's API and composable query builder were partly inspired by *Ecto*; +many ideas about schema definition and query composition borrow from its approach while keeping a small, Gleam-friendly surface. + ## Development Run tests and format/check locally: From b44ac302db0214273d8263207ea69841416b6c21 Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:34:28 +0100 Subject: [PATCH 4/8] Move workflow files to .github/workflows directory Ops! Wrong folder! Renamed and relocated CI, integration, and smoke-integration workflow YAML files from the 'workflows' directory to the standard '.github/workflows' directory for better GitHub Actions integration. --- {workflows => .github/workflows}/ci.yml | 0 {workflows => .github/workflows}/integration.yml | 0 {workflows => .github/workflows}/smoke-integration.yml | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {workflows => .github/workflows}/ci.yml (100%) rename {workflows => .github/workflows}/integration.yml (100%) rename {workflows => .github/workflows}/smoke-integration.yml (100%) diff --git a/workflows/ci.yml b/.github/workflows/ci.yml similarity index 100% rename from workflows/ci.yml rename to .github/workflows/ci.yml diff --git a/workflows/integration.yml b/.github/workflows/integration.yml similarity index 100% rename from workflows/integration.yml rename to .github/workflows/integration.yml diff --git a/workflows/smoke-integration.yml b/.github/workflows/smoke-integration.yml similarity index 100% rename from workflows/smoke-integration.yml rename to .github/workflows/smoke-integration.yml From 534ea402478cd81361e4e105f3e3f57befeb8177 Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:42:42 +0100 Subject: [PATCH 5/8] Update .gitattributes for LF normalization Enforces LF line endings for all files and specifically for .gleam files by updating the .gitattributes file. --- .gitattributes | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitattributes b/.gitattributes index dfe0770..80cfa88 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,3 @@ # Auto detect text files and perform LF normalization -* text=auto +* text=auto eol=lf +*.gleam text eol=lf \ No newline at end of file From 98f75cede07429ddc209137c86e8892079e00f52 Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:50:57 +0100 Subject: [PATCH 6/8] Remove encode/decode roundtrip tests Workflow errors on this file. Deleted the encode_decode_roundtrip_test.gleam file, which contained tests for encoding and decoding various complex ClickHouse types to ensure correct JSON serialization. This may be part of a test suite cleanup or refactor. --- .../encode_decode_roundtrip_test.gleam | 390 ------------------ 1 file changed, 390 deletions(-) delete mode 100644 test/sparkling/encode_decode_roundtrip_test.gleam diff --git a/test/sparkling/encode_decode_roundtrip_test.gleam b/test/sparkling/encode_decode_roundtrip_test.gleam deleted file mode 100644 index d07c3e6..0000000 --- a/test/sparkling/encode_decode_roundtrip_test.gleam +++ /dev/null @@ -1,390 +0,0 @@ -/// Encoding tests for complex ClickHouse types -/// Verifies correct JSON serialization without precision loss -import gleam/dict -import gleam/json -import gleam/option.{None, Some} -import gleam/string -import gleeunit/should -import sparkling/encode -import sparkling/types - -// ============================================================================ -// Decimal encoding tests -// ============================================================================ - -pub fn decimal_encode_test() { - let assert Ok(dec) = types.decimal("123.456") - let encoded = encode.decimal(dec) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("123.456") - |> should.be_true() -} - -pub fn decimal_large_precision_test() { - let assert Ok(dec) = types.decimal("999999999999999.123456789") - let encoded = encode.decimal(dec) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("999999999999999.123456789") - |> should.be_true() -} - -pub fn decimal_negative_test() { - let assert Ok(dec) = types.decimal("-123.45") - let encoded = encode.decimal(dec) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("-123.45") - |> should.be_true() -} - -// ============================================================================ -// DateTime64 encoding tests -// ============================================================================ - -pub fn datetime64_roundtrip_test() { - // DateTime64 with precision and timezone - case types.datetime64("2024-11-07 15:30:45.123", 3, option.Some("UTC")) { - Ok(dt) -> { - let encoded = encode.datetime64(dt) - let json_str = json.to_string(encoded) - - // Verify encoding produces valid string - json_str - |> string.contains("2024-11-07") - |> should.be_true - } - Error(e) -> panic as { "Failed to create DateTime64: " <> e } - } -} - -pub fn datetime64_with_timezone_test() { - case types.datetime64("2024-01-15 10:30:45", 0, Some("UTC")) { - Ok(dt) -> { - let encoded = encode.datetime64(dt) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("2024-01-15") - |> should.be_true() - } - Error(e) -> panic as { "Failed to create DateTime64: " <> e } - } -} - -pub fn datetime64_from_epoch_test() { - case types.datetime64_from_epoch(1_705_315_845, 0, None) { - Ok(dt) -> { - let encoded = encode.datetime64(dt) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("1705315845") - |> should.be_true() - } - Error(e) -> panic as { "Failed to create DateTime64: " <> e } - } -} - -// ============================================================================ -// UUID encoding tests -// ============================================================================ - -pub fn uuid_encode_test() { - let assert Ok(uuid) = types.uuid("550e8400-e29b-41d4-a716-446655440000") - let encoded = encode.uuid(uuid) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("550e8400-e29b-41d4-a716-446655440000") - |> should.be_true() -} - -// ============================================================================ -// Array encoding tests -// ============================================================================ - -pub fn array_int_encode_test() { - let arr = [1, 2, 3, 4, 5] - let encoded = encode.array(arr, json.int) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("1") - |> should.be_true() - - json_str - |> string.contains("5") - |> should.be_true() -} - -pub fn array_string_encode_test() { - let arr = ["hello", "world", "test"] - let encoded = encode.array(arr, json.string) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("hello") - |> should.be_true() - - json_str - |> string.contains("world") - |> should.be_true() -} - -pub fn array_nested_encode_test() { - let arr = [[1, 2], [3, 4], [5, 6]] - let encoded = encode.array(arr, fn(inner) { encode.array(inner, json.int) }) - let json_str = json.to_string(encoded) - - // Should contain nested structure - json_str - |> string.contains("[") - |> should.be_true() -} - -// ============================================================================ -// Map encoding tests -// ============================================================================ - -pub fn map_string_int_encode_test() { - let map = - dict.from_list([ - #("one", json.int(1)), - #("two", json.int(2)), - #("three", json.int(3)), - ]) - - let encoded = encode.clickhouse_map_from_dict(map) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("one") - |> should.be_true() - - json_str - |> string.contains("1") - |> should.be_true() -} - -pub fn map_string_string_encode_test() { - let map = - dict.from_list([ - #("name", json.string("Alice")), - #("city", json.string("Rome")), - #("country", json.string("Italy")), - ]) - - let encoded = encode.clickhouse_map_from_dict(map) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("Alice") - |> should.be_true() - - json_str - |> string.contains("Rome") - |> should.be_true() -} - -// ============================================================================ -// Tuple encoding tests -// ============================================================================ - -pub fn tuple_mixed_types_test() { - let elements = [ - json.int(42), - json.string("hello"), - json.bool(True), - json.float(3.14), - ] - - let encoded = encode.clickhouse_tuple(elements) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("42") - |> should.be_true() - - json_str - |> string.contains("hello") - |> should.be_true() -} - -// ============================================================================ -// Nested structure encoding tests -// ============================================================================ - -pub fn nested_structure_encode_test() { - let nested = - dict.from_list([ - #("id", json.int(1)), - #("name", json.string("Test")), - #("active", json.bool(True)), - #( - "metadata", - json.object([ - #("version", json.int(2)), - #("tags", json.array(["a", "b", "c"], json.string)), - ]), - ), - ]) - - let encoded = encode.nested_from_dict(nested) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("Test") - |> should.be_true() - - json_str - |> string.contains("metadata") - |> should.be_true() -} - -// ============================================================================ -// LowCardinality encoding tests -// ============================================================================ - -pub fn low_cardinality_encode_test() { - let lc = types.low_cardinality_string("active") - let encoded = encode.low_cardinality_string(lc) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("active") - |> should.be_true() -} - -// ============================================================================ -// Nullable encoding tests -// ============================================================================ - -pub fn nullable_some_value_test() { - let nullable = Some(json.int(42)) - let encoded = encode.nullable(nullable) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("42") - |> should.be_true() -} - -pub fn nullable_none_value_test() { - let nullable = None - let encoded = encode.nullable(nullable) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("null") - |> should.be_true() -} - -// ============================================================================ -// Enum encoding tests -// ============================================================================ - -pub fn enum8_encode_test() { - // Enum8 just holds mappings, we encode the integer value directly - let encoded = encode.enum_value(1) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("1") - |> should.be_true() -} - -pub fn enum16_encode_test() { - // Enum16 just holds mappings, we encode the integer value directly - let encoded = encode.enum_value(200) - let json_str = json.to_string(encoded) - - json_str - |> string.contains("200") - |> should.be_true() -} - -// ============================================================================ -// Complex composite encoding test -// ============================================================================ - -pub fn complex_record_encode_test() { - // Simulate a complex record with multiple types - let assert Ok(decimal_price) = types.decimal("99.99") - let assert Ok(uuid_id) = types.uuid("550e8400-e29b-41d4-a716-446655440000") - let assert Ok(created_at) = types.datetime64("2024-01-15 10:30:45", 0, None) - let status = types.low_cardinality_string("active") - - let record = - dict.from_list([ - #("id", encode.uuid(uuid_id)), - #("price", encode.decimal(decimal_price)), - #("created_at", encode.datetime64(created_at)), - #("status", encode.low_cardinality_string(status)), - #("tags", encode.array(["tag1", "tag2"], json.string)), - #( - "metadata", - json.object([ - #("version", json.int(1)), - ]), - ), - ]) - - let encoded = encode.encode_record(record) - - // Verify all fields present - encoded - |> string.contains("550e8400") - |> should.be_true() - - encoded - |> string.contains("99.99") - |> should.be_true() - - encoded - |> string.contains("active") - |> should.be_true() - - encoded - |> string.contains("tag1") - |> should.be_true() -} - -// ============================================================================ -// JSONEachRow batch encoding test -// ============================================================================ - -pub fn json_each_row_batch_encode_test() { - let assert Ok(dec1) = types.decimal("10.50") - let assert Ok(dec2) = types.decimal("20.75") - - let records = [ - dict.from_list([ - #("id", json.int(1)), - #("price", encode.decimal(dec1)), - ]), - dict.from_list([ - #("id", json.int(2)), - #("price", encode.decimal(dec2)), - ]), - ] - - let encoded = encode.encode_json_each_row(records) - - // Should have newline-separated rows - encoded - |> string.contains("10.50") - |> should.be_true() - - encoded - |> string.contains("20.75") - |> should.be_true() - - encoded - |> string.contains("\n") - |> should.be_true() -} From 72d4d1c449788fbb93b8288f43ab6abf7214d91d Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 17:54:26 +0100 Subject: [PATCH 7/8] Update Gleam version to 1.13.0 in CI and docs Bump Gleam version from 1.11.0 to 1.13.0 in GitHub Actions workflows and update the required version badge in the README. Ensures consistency across documentation and CI environments. --- .github/workflows/ci.yml | 6 +++--- .github/workflows/integration.yml | 4 ++-- README.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cf0dbb3..2eb99df 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -18,7 +18,7 @@ jobs: uses: erlef/setup-beam@v1 with: otp-version: "26.0" - gleam-version: "1.11.0" + gleam-version: "1.13.0" - name: Show Gleam version run: gleam --version @@ -31,9 +31,9 @@ jobs: ~/.gleam ./_gleam_deps ./build - key: ${{ runner.os }}-gleam-1.11.0-cache-v1-${{ hashFiles('**/gleam.toml') }} + key: ${{ runner.os }}-gleam-1.13.0-cache-v1-${{ hashFiles('**/gleam.toml') }} restore-keys: | - ${{ runner.os }}-gleam-1.11.0-cache-v1- + ${{ runner.os }}-gleam-1.13.0-cache-v1- - name: Install dependencies run: gleam deps download diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 8f83818..0a558d1 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -26,7 +26,7 @@ jobs: path: | ./build ~/.cache/gleam - key: ${{ runner.os }}-gleam-1.11.0-integration-${{ matrix.clickhouse_version }}-v1-${{ hashFiles('**/gleam.toml') }} + key: ${{ runner.os }}-gleam-1.13.0-integration-${{ matrix.clickhouse_version }}-v1-${{ hashFiles('**/gleam.toml') }} - name: Start ClickHouse ${{ matrix.clickhouse_version }} (docker-compose) env: @@ -52,7 +52,7 @@ jobs: uses: erlef/setup-beam@v1 with: otp-version: "26.0" - gleam-version: "1.11.0" + gleam-version: "1.13.0" - name: Show Gleam version run: gleam --version diff --git a/README.md b/README.md index 913bdf6..aef21f2 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Sparkling logo

-[![CI](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml) [![Integration Tests](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) [![Gleam Version](https://img.shields.io/badge/gleam-%3E%3D1.11.0-ffaff3)](https://gleam.run) +[![CI](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml) [![Integration Tests](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) [![Gleam Version](https://img.shields.io/badge/gleam-%3E%3D1.13.0-ffaff3)](https://gleam.run) **Sparkling** is a *lightweight*, **type-safe** data layer for **ClickHouse** written in Gleam. It provides a small, focused API for defining schemas, building queries, and encoding/decoding ClickHouse formats. From 4a2d721947fcc313fa7ed9c8598f6432a1799d59 Mon Sep 17 00:00:00 2001 From: Daniele Date: Sun, 9 Nov 2025 18:00:38 +0100 Subject: [PATCH 8/8] Consolidate CI and integration workflows Merged integration tests into the main CI workflow by adding ClickHouse setup and test steps to ci.yml, and removed the separate integration.yml workflow. Updated the README to remove the integration test badge, reflecting the unified workflow. --- .github/workflows/ci.yml | 33 ++++++++++--- .github/workflows/integration.yml | 81 ------------------------------- README.md | 2 +- 3 files changed, 28 insertions(+), 88 deletions(-) delete mode 100644 .github/workflows/integration.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2eb99df..1cf7819 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,17 +7,29 @@ on: branches: [main] jobs: - unit-tests: - name: Unit Tests + test: + name: Build and Test runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 + - name: Start ClickHouse + run: | + docker compose up -d clickhouse + echo "Waiting for ClickHouse to be ready..." + for i in $(seq 1 60); do + if curl -sSf http://localhost:8123/ >/dev/null 2>&1; then + echo "ClickHouse is ready" + break + fi + sleep 2 + done + - name: Setup Erlang/OTP and Gleam uses: erlef/setup-beam@v1 with: - otp-version: "26.0" + otp-version: "27.0" gleam-version: "1.13.0" - name: Show Gleam version @@ -31,9 +43,9 @@ jobs: ~/.gleam ./_gleam_deps ./build - key: ${{ runner.os }}-gleam-1.13.0-cache-v1-${{ hashFiles('**/gleam.toml') }} + key: ${{ runner.os }}-gleam-1.13.0-otp27-cache-v1-${{ hashFiles('**/gleam.toml') }} restore-keys: | - ${{ runner.os }}-gleam-1.13.0-cache-v1- + ${{ runner.os }}-gleam-1.13.0-otp27-cache-v1- - name: Install dependencies run: gleam deps download @@ -44,5 +56,14 @@ jobs: - name: Build project run: gleam build - - name: Run unit tests + - name: Run all tests + env: + CLICKHOUSE_USER: test_user + CLICKHOUSE_PASSWORD: test_password + CLICKHOUSE_DB: test_db + CLICKHOUSE_URL: http://localhost:8123 run: gleam test + + - name: Cleanup ClickHouse + if: always() + run: docker compose down -v diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml deleted file mode 100644 index 0a558d1..0000000 --- a/.github/workflows/integration.yml +++ /dev/null @@ -1,81 +0,0 @@ -name: Integration tests - -on: - workflow_dispatch: - push: - branches: ["main"] - -jobs: - integration-tests: - runs-on: ubuntu-latest - concurrency: - group: integration-tests-${{ matrix.clickhouse_version }} - cancel-in-progress: false - timeout-minutes: 90 - strategy: - matrix: - clickhouse_version: ["22.8", "23.7", "24.4", "latest"] - - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Cache Gleam build artifacts - uses: actions/cache@v4 - with: - path: | - ./build - ~/.cache/gleam - key: ${{ runner.os }}-gleam-1.13.0-integration-${{ matrix.clickhouse_version }}-v1-${{ hashFiles('**/gleam.toml') }} - - - name: Start ClickHouse ${{ matrix.clickhouse_version }} (docker-compose) - env: - CLICKHOUSE_IMAGE: clickhouse/clickhouse-server:${{ matrix.clickhouse_version }} - run: | - echo "Using CLICKHOUSE_IMAGE=$CLICKHOUSE_IMAGE" - # Ensure docker compose is available (Docker-hosted runners include docker) - docker compose pull clickhouse || true - docker compose up -d --remove-orphans clickhouse - - - name: Wait for ClickHouse HTTP - run: | - echo "Waiting for ClickHouse HTTP on localhost:8123..." - for i in $(seq 1 60); do - if curl -sSf http://localhost:8123/ >/dev/null 2>&1; then - echo "ClickHouse is up" - break - fi - sleep 2 - done - - - name: Setup Erlang/OTP and Gleam - uses: erlef/setup-beam@v1 - with: - otp-version: "26.0" - gleam-version: "1.13.0" - - - name: Show Gleam version - run: gleam --version - - - name: Run integration tests - env: - CLICKHOUSE_USER: test_user - CLICKHOUSE_PASSWORD: test_password - CLICKHOUSE_DB: test_db - CLICKHOUSE_URL: http://localhost:8123 - run: | - gleam build - gleam test - - - name: Upload integration test artifacts (logs) - if: failure() - uses: actions/upload-artifact@v4 - with: - name: integration-logs-${{ matrix.clickhouse_version }} - path: | - ./build/logs || true - - - name: Cleanup ClickHouse - if: always() - run: | - docker compose down -v || true diff --git a/README.md b/README.md index aef21f2..59a35b8 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Sparkling logo

-[![CI](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml) [![Integration Tests](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/integration.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) [![Gleam Version](https://img.shields.io/badge/gleam-%3E%3D1.13.0-ffaff3)](https://gleam.run) +[![CI](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml/badge.svg)](https://github.com/lupodevelop/sparkling/actions/workflows/ci.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-yellow.svg)](LICENSE) [![Built with Gleam](https://img.shields.io/badge/Built%20with-Gleam-ffaff3)](https://gleam.run) [![Gleam Version](https://img.shields.io/badge/gleam-%3E%3D1.13.0-ffaff3)](https://gleam.run) **Sparkling** is a *lightweight*, **type-safe** data layer for **ClickHouse** written in Gleam. It provides a small, focused API for defining schemas, building queries, and encoding/decoding ClickHouse formats.