diff --git a/docs/admin/troubleshooting/jcmd/docker.md b/docs/admin/troubleshooting/jcmd/docker.md index eb6c58f2..af09a0da 100644 --- a/docs/admin/troubleshooting/jcmd/docker.md +++ b/docs/admin/troubleshooting/jcmd/docker.md @@ -64,7 +64,7 @@ Let's start by starting a `cratedb` container. ```console docker run --rm -it --name=cratedb \ --publish=4200:4200 --publish=5432:5432 \ - --env=CRATE_HEAP_SIZE=2g crate/crate:nightly \ + --env=CRATE_HEAP_SIZE=1g crate/crate:nightly \ -Cdiscovery.type=single-node ``` diff --git a/docs/feature/search/fts/analyzer.md b/docs/feature/search/fts/analyzer.md index b2133dd6..fd6fcec8 100644 --- a/docs/feature/search/fts/analyzer.md +++ b/docs/feature/search/fts/analyzer.md @@ -165,10 +165,10 @@ s(900516492,3,'Bill',n,1,0). ``` ```shell -docker run --rm -it --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \ +docker run --rm -it --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=1g \ --volume="$PWD/synonyms-solr.txt:/crate/config/synonyms-solr.txt" \ --volume="$PWD/synonyms-wordnet.txt:/crate/config/synonyms-wordnet.txt" \ - crate -Cdiscovery.type=single-node + crate ``` This example uses the `synonyms-solr.txt` in Solr format. diff --git a/docs/install/container/docker.rst b/docs/install/container/docker.rst index 94fd80d1..f68fb671 100644 --- a/docs/install/container/docker.rst +++ b/docs/install/container/docker.rst @@ -83,7 +83,7 @@ Breaking the command down: - Puts the container into the ``crate`` network and maps port ``4201`` on your host machine to port ``4200`` on the container (admin UI). - Defines the environment variable:ref:`CRATE_HEAP_SIZE `, - which is used by CrateDB to allocate 2G for its heap. + which is used by CrateDB to allocate 1 GB for its heap memory. - Runs the command ``crate`` inside the container with parameters: * ``network.host``: The ``_site_`` value results in the binding of the CrateDB process to a site-local IP address. @@ -298,7 +298,7 @@ define your services like this: restart_policy: condition: on-failure environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g cratedb02: image: crate:latest @@ -320,7 +320,7 @@ define your services like this: restart_policy: condition: on-failure environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g cratedb03: image: crate:latest @@ -342,7 +342,7 @@ define your services like this: restart_policy: condition: on-failure environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g In the file above: @@ -485,7 +485,7 @@ example:: $ docker run -d \ --cpus 1.5 \ - --memory 2g \ + --memory 1g \ --env CRATE_HEAP_SIZE=1g \ crate \ crate -Cnetwork.host=_site_ diff --git a/docs/integrate/airflow/data-retention-hot-cold.md b/docs/integrate/airflow/data-retention-hot-cold.md index 1c2512dd..7892f53a 100644 --- a/docs/integrate/airflow/data-retention-hot-cold.md +++ b/docs/integrate/airflow/data-retention-hot-cold.md @@ -34,7 +34,7 @@ services: "-Cgateway.expected_data_nodes=3", "-Cgateway.recover_after_data_nodes=2"] environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g cratedb02: image: crate:latest @@ -53,7 +53,7 @@ services: "-Cgateway.expected_data_nodes=3", "-Cgateway.recover_after_data_nodes=2"] environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g cratedb03: image: crate:latest @@ -72,7 +72,7 @@ services: "-Cgateway.expected_data_nodes=3", "-Cgateway.recover_after_data_nodes=2"] environment: - - CRATE_HEAP_SIZE=2g + - CRATE_HEAP_SIZE=1g ``` Start the cluster with `docker compose up`. For details, see the linked documentation. diff --git a/docs/integrate/airflow/getting-started.md b/docs/integrate/airflow/getting-started.md index 6e589766..cfad558c 100644 --- a/docs/integrate/airflow/getting-started.md +++ b/docs/integrate/airflow/getting-started.md @@ -17,7 +17,7 @@ You will: :::{rubric} Apache Airflow ::: Apache Airflow programmatically creates, schedules, and monitors workflows -\[[Official documentation](https://airflow.apache.org/docs/)\]. A workflow +([Official documentation](https://airflow.apache.org/docs/)). A workflow is a directed acyclic graph (DAG) where each node represents a task. Each task runs independently; the DAG tracks dependencies. Run DAGs on demand or on schedules (for example, twice a week). diff --git a/docs/integrate/amqp/compose.yaml b/docs/integrate/amqp/compose.yaml new file mode 100644 index 00000000..3cdaa1ed --- /dev/null +++ b/docs/integrate/amqp/compose.yaml @@ -0,0 +1,28 @@ +# Service composition file for Docker Compose or Podman Compose +# https://cratedb.com/docs/guide/integrate/amqp/ + +services: + + cratedb: + image: docker.io/crate/crate:latest + ports: + - "4200:4200/tcp" + - "5432:5432/tcp" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:4200/"] + interval: 5s + timeout: 30s + retries: 5 + + rabbitmq: + image: docker.io/rabbitmq:3 + ports: + - "5672:5672/tcp" + + lorrystream: + image: ghcr.io/daq-tools/lorrystream:latest + profiles: ["cli"] + + amqpcat: + image: docker.io/cloudamqp/amqpcat:latest + profiles: ["cli"] diff --git a/docs/integrate/amqp/usage.md b/docs/integrate/amqp/usage.md index 7999570c..3f2496d0 100644 --- a/docs/integrate/amqp/usage.md +++ b/docs/integrate/amqp/usage.md @@ -11,84 +11,60 @@ pipeline element. ## Prerequisites -Docker is used for running all components. This approach works consistently +Use Docker or Podman to run all components. This approach works consistently across Linux, macOS, and Windows. -Alternatively, you can use Podman. You can also use a different AMQP broker such as -Apache Qpid, Apache ActiveMQ, IBM MQ, or Solace. Azure Event Hubs and Azure Service -Bus speak AMQP as well, but with protocol and authentication specifics; adjust -settings accordingly. +### Files -Create a shared network. -```shell -docker network create cratedb-demo -``` +First, download and save all required files to your machine. +- {download}`compose.yaml` -Start CrateDB. -```shell -docker run --name=cratedb --rm --network=cratedb-demo \ - --publish=4200:4200 --publish=5432:5432 \ - --env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node -``` +### Services -Start RabbitMQ. -```shell -docker run --name=rabbitmq --rm --network=cratedb-demo \ - --publish=5672:5672 docker.io/rabbitmq:3 -``` -> Note: This broker configuration allows anonymous access for demonstration purposes only. -> Do not expose it to untrusted networks. For production, configure authentication/TLS. +Start services using Docker Compose or Podman Compose. +If you use Podman, replace `docker` with `podman` (or enable the podman‑docker +compatibility shim) and run `podman compose up`. -Prepare shortcuts for the CrateDB shell, LorryStream, and the AMQP client -programs. - -::::{tab-set} +You can also use a different AMQP broker such as +Apache Qpid, Apache ActiveMQ, IBM MQ, or Solace. Azure Event Hubs and Azure Service +Bus speak AMQP as well, but with protocol and authentication specifics; adjust +settings accordingly. -:::{tab-item} Linux and macOS -To make the settings persistent, add them to your shell profile (e.g., `~/.profile` or `~/.zshrc`). ```shell -alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" -alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry" -alias amqpcat="docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat" -``` -::: -:::{tab-item} Windows PowerShell -To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). -```powershell -function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } -function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args } -function amqpcat { docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat @args } +docker compose up ``` -::: -:::{tab-item} Windows Command -```shell -doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* -doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $* -doskey amqpcat=docker run --rm -i --network=cratedb-demo docker.io/cloudamqp/amqpcat amqpcat $* -``` -::: -:::: +:::{note} +The AMQP broker configuration used here allows anonymous access for +demonstration purposes only. Do not expose it to untrusted networks. For +production, configure authentication/TLS. +::: -## Usage +## Submit data Invoke the data transfer pipeline. ```shell -lorry relay \ - "amqp://guest:guest@rabbitmq:5672/%2F?exchange=default&queue=default&routing-key=testdrive&setup=exchange,queue,bind&content-type=json" \ - "crate://cratedb/?table=testdrive" +docker compose run --rm --no-TTY lorrystream lorry relay "amqp://guest:guest@rabbitmq:5672/%2F?exchange=default&queue=default&routing-key=testdrive&setup=exchange,queue,bind&content-type=json" "crate://cratedb/?table=amqp_demo" ``` Publish a JSON message to AMQP. ```shell -echo '{"temperature": 42.84, "humidity": 83.1}' | \ - amqpcat --producer --uri='amqp://guest:guest@rabbitmq:5672/%2F' \ - --exchange=default --queue=default --routing-key=testdrive +echo '{"temperature": 42.84, "humidity": 83.1}' | docker compose run --rm --no-TTY amqpcat amqpcat --producer --uri='amqp://guest:guest@rabbitmq:5672/%2F' --exchange=default --queue=default --routing-key=testdrive ``` +## Explore data + Inspect data stored in CrateDB. ```shell -crash --hosts cratedb -c "SELECT * FROM testdrive" +docker compose exec cratedb crash -c "SELECT * FROM doc.amqp_demo" +``` +```psql ++-------------+----------+ +| temperature | humidity | ++-------------+----------+ +| 42.84 | 83.1 | ++-------------+----------+ +SELECT 1 row in set (0.023 sec) ``` diff --git a/docs/integrate/collectd/Dockerfile b/docs/integrate/collectd/Dockerfile index 4001a460..fd5d451e 100644 --- a/docs/integrate/collectd/Dockerfile +++ b/docs/integrate/collectd/Dockerfile @@ -1,3 +1,5 @@ +# https://cratedb.com/docs/guide/integrate/collectd/ + FROM docker.io/debian:13-slim # Configure system environment. diff --git a/docs/integrate/collectd/collectd-cratedb.conf b/docs/integrate/collectd/collectd-cratedb.conf index ecc314f0..b014e43f 100644 --- a/docs/integrate/collectd/collectd-cratedb.conf +++ b/docs/integrate/collectd/collectd-cratedb.conf @@ -1,4 +1,6 @@ # collectd configuration for storing metrics into CrateDB. +# https://cratedb.com/docs/guide/integrate/collectd/ + # https://collectd.org/documentation/manpages/collectd.conf.html#plugin-postgresql LoadPlugin postgresql diff --git a/docs/integrate/collectd/collectd-telegraf.conf b/docs/integrate/collectd/collectd-telegraf.conf index edabe539..b766e1d1 100644 --- a/docs/integrate/collectd/collectd-telegraf.conf +++ b/docs/integrate/collectd/collectd-telegraf.conf @@ -1,4 +1,6 @@ # collectd configuration for sending metrics to Telegraf. +# https://cratedb.com/docs/guide/integrate/collectd/ + # https://collectd.org/documentation/manpages/collectd.conf.html#plugin-network LoadPlugin network diff --git a/docs/integrate/collectd/compose.yaml b/docs/integrate/collectd/compose.yaml new file mode 100644 index 00000000..3600881e --- /dev/null +++ b/docs/integrate/collectd/compose.yaml @@ -0,0 +1,42 @@ +# Service composition file for Docker Compose or Podman Compose +# https://cratedb.com/docs/guide/integrate/collectd/ + +services: + + cratedb: + image: docker.io/crate/crate:latest + ports: + - "4200:4200/tcp" + - "5432:5432/tcp" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:4200/"] + interval: 5s + timeout: 30s + retries: 5 + + collectd-cratedb: + build: . + volumes: + - ./collectd-cratedb.conf:/etc/collectd/collectd.conf.d/collectd-cratedb.conf:ro + depends_on: + - cratedb + + collectd-telegraf: + build: . + volumes: + - ./collectd-telegraf.conf:/etc/collectd/collectd.conf.d/collectd-telegraf.conf:ro + depends_on: + - cratedb + + telegraf: + image: docker.io/telegraf:latest + ports: + - "8125:8125/udp" # StatsD + volumes: + - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro + depends_on: + - cratedb + + postgresql: + image: docker.io/postgres:18 + profiles: ["cli"] diff --git a/docs/integrate/collectd/telegraf.conf b/docs/integrate/collectd/telegraf.conf index 08cddb27..125d61c4 100644 --- a/docs/integrate/collectd/telegraf.conf +++ b/docs/integrate/collectd/telegraf.conf @@ -1,3 +1,6 @@ +# Telegraf configuration with CrateDB output. +# https://cratedb.com/docs/guide/integrate/collectd/ + [[inputs.socket_listener]] service_address = "udp://:25826" diff --git a/docs/integrate/collectd/usage-collectd.md b/docs/integrate/collectd/usage-collectd.md index 2ed7a512..f9a7502a 100644 --- a/docs/integrate/collectd/usage-collectd.md +++ b/docs/integrate/collectd/usage-collectd.md @@ -6,60 +6,39 @@ so that collectd sends system metrics and CrateDB stores them. ## Prerequisites -Docker runs all components consistently across Linux, macOS, and Windows. -If you use Podman, substitute podman for docker in the commands. +Use Docker or Podman to run all components. This approach works consistently +across Linux, macOS, and Windows. -### Commands +### Files -Prepare shortcut for the psql command. +First, download and save all required files to your machine. +- {download}`compose.yaml` +- {download}`Dockerfile` +- {download}`collectd-cratedb.conf` -::::{tab-set} -:sync-group: os +### Services -:::{tab-item} Linux and macOS -:sync: unix -To make the settings persistent, add them to your shell profile (`~/.profile`). -```shell -alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres:16 psql" -``` -::: -:::{tab-item} Windows PowerShell -:sync: powershell -To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). -```powershell -function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres:16 psql @args } -``` -::: -:::{tab-item} Windows Command -:sync: dos -```shell -doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres:16 psql $* -``` -::: - -:::: +Start services using Docker Compose or Podman Compose. +If you use Podman, replace `docker` with `podman` (or enable the podman‑docker +compatibility shim) and run `podman compose up`. -### CrateDB - -Create a shared network. ```shell -docker network create cratedb-demo +docker compose up ``` -Start CrateDB. -```shell -docker run --name=cratedb --rm -it --network=cratedb-demo \ - --publish=4200:4200 --publish=5432:5432 \ - --env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node -``` +To send the collected data to CrateDB, collectd is configured to load its +[`postgresql` plugin]. -## Configure +::::{dropdown} collectd configuration `collectd-cratedb.conf` +:::{literalinclude} collectd-cratedb.conf +::: +:::: ### Provision database Create a database table that stores collected metrics. ```shell -psql "postgresql://crate:crate@cratedb:5432/" < + /usr/sbin/mosquitto -c /mosquitto-no-auth.conf + ports: + - "1883:1883/tcp" + + lorrystream: + image: ghcr.io/daq-tools/lorrystream:latest + profiles: ["cli"] diff --git a/docs/integrate/mqtt/usage.md b/docs/integrate/mqtt/usage.md index 194845b8..7e05ce80 100644 --- a/docs/integrate/mqtt/usage.md +++ b/docs/integrate/mqtt/usage.md @@ -10,93 +10,67 @@ pipeline element. ## Prerequisites -Docker is used for running all components. This approach works consistently +Use Docker or Podman to run all components. This approach works consistently across Linux, macOS, and Windows. -Alternatively, you can use Podman. You can also use a different MQTT broker such as +You can also use a different MQTT broker such as EMQX, HiveMQ, VerneMQ, or RabbitMQ. Azure IoT Hub speaks MQTT as well, but with protocol and authentication specifics; adjust settings accordingly. -Create a shared network. -```shell -docker network create cratedb-demo -``` - -Start CrateDB. -```shell -docker run --name=cratedb --rm -it --network=cratedb-demo \ - --publish=4200:4200 --publish=5432:5432 \ - --env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node -``` +### Files -Start Mosquitto. -```shell -docker run --name=mosquitto --rm -it --network=cratedb-demo \ - --publish=1883:1883 docker.io/eclipse-mosquitto \ - mosquitto -c /mosquitto-no-auth.conf -``` -> Note: This broker configuration allows anonymous access for demonstration purposes only. -> Do not expose it to untrusted networks. For production, configure authentication/TLS. +First, download and save all required files to your machine. +- {download}`compose.yaml` -Prepare shortcuts for the CrateDB shell, LorryStream, and the Mosquitto client -programs. +### Services -::::{tab-set} +Start services using Docker Compose or Podman Compose. +If you use Podman, replace `docker` with `podman` (or enable the podman‑docker +compatibility shim) and run `podman compose up`. -:::{tab-item} Linux and macOS -To make the settings persistent, add them to your shell profile (`~/.profile`). -```shell -alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" -alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry" -alias mosquitto_pub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub" -alias mosquitto_sub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub" -``` -::: -:::{tab-item} Windows PowerShell -To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). -```powershell -function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } -function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args } -function mosquitto_pub { docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub @args } -function mosquitto_sub { docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub @args } -``` -::: -:::{tab-item} Windows Command ```shell -doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* -doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $* -doskey mosquitto_pub=docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub $* -doskey mosquitto_sub=docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub $* +docker compose up ``` -::: -:::: +:::{note} +The MQTT broker configuration used here allows anonymous access for +demonstration purposes only. Do not expose it to untrusted networks. For +production, configure authentication/TLS. +::: -## Usage +## Submit data Subscribe to all MQTT topics on the broker to monitor any traffic. ```shell -mosquitto_sub -h mosquitto -t "#" -v +docker compose exec --no-tty mosquitto mosquitto_sub -h mosquitto -t "#" -v ``` Invoke the data transfer pipeline. ```shell -lorry relay \ - "mqtt://mosquitto/testdrive/%23?content-type=json" \ - "crate://cratedb/?table=testdrive" +docker compose run --rm lorrystream lorry relay "mqtt://mosquitto/testdrive/%23?content-type=json" "crate://cratedb/?table=mqtt_demo" ``` Publish a JSON message to an MQTT topic. ```shell -echo '{"temperature": 42.84, "humidity": 83.1}' | \ - mosquitto_pub -h mosquitto -t testdrive/channel1 -s +echo '{"temperature": 42.84, "humidity": 83.1}' | docker compose exec --no-tty mosquitto mosquitto_pub -h mosquitto -t testdrive/channel1 -s ``` +## Explore data + Inspect data stored in CrateDB. ```shell -crash --hosts cratedb -c "SELECT * FROM testdrive" +docker compose exec cratedb crash -c "SELECT * FROM doc.mqtt_demo" +``` +```psql ++-------------+----------+ +| temperature | humidity | ++-------------+----------+ +| 42.84 | 83.1 | ++-------------+----------+ +SELECT 1 row in set (0.004 sec) ``` [Eclipse Mosquitto]: https://mosquitto.org/ +[LorryStream]: https://lorrystream.readthedocs.io/ [LorryStream MQTT source]: https://lorrystream.readthedocs.io/source/mqtt.html diff --git a/docs/integrate/mysql/compose.yaml b/docs/integrate/mysql/compose.yaml new file mode 100644 index 00000000..9734efef --- /dev/null +++ b/docs/integrate/mysql/compose.yaml @@ -0,0 +1,26 @@ +# Service composition file for Docker Compose or Podman Compose +# https://cratedb.com/docs/guide/integrate/mysql/ + +services: + + cratedb: + image: docker.io/crate/crate:latest + ports: + - "4200:4200/tcp" + - "5432:5432/tcp" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:4200/"] + interval: 5s + timeout: 30s + retries: 5 + + mariadb: + image: docker.io/mariadb:latest + environment: + MARIADB_ROOT_PASSWORD: secret + ports: + - "3306:3306/tcp" + + ctk-ingest: + image: ghcr.io/crate/cratedb-toolkit-ingest:latest + profiles: ["cli"] diff --git a/docs/integrate/mysql/usage.md b/docs/integrate/mysql/usage.md index 3dae3c0b..c78f64ef 100644 --- a/docs/integrate/mysql/usage.md +++ b/docs/integrate/mysql/usage.md @@ -10,67 +10,32 @@ The data transfer is supported by the ## Prerequisites -Docker is used for running all components. This approach works consistently -across Linux, macOS, and Windows. Alternatively, you can use Podman. +Use Docker or Podman to run all components. This approach works consistently +across Linux, macOS, and Windows. -Create a shared network. -```shell -docker network create cratedb-demo -``` - -Start CrateDB. -```shell -docker run --name=cratedb --rm --network=cratedb-demo \ - --publish=4200:4200 --publish=5432:5432 --env=CRATE_HEAP_SIZE=2g \ - docker.io/crate -Cdiscovery.type=single-node -``` +### Files -Start MariaDB. -```shell -docker run --name=mariadb --rm --network=cratedb-demo \ - --publish=3306:3306 --env "MARIADB_ROOT_PASSWORD=secret" \ - docker.io/mariadb -``` +First, download and save all required files to your machine. +- {download}`compose.yaml` -Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the MariaDB client -programs. +### Services -::::{tab-set} +Start services using Docker Compose or Podman Compose. +If you use Podman, replace `docker` with `podman` (or enable the podman‑docker +compatibility shim) and run `podman compose up`. -:::{tab-item} Linux and macOS -To make the settings persistent, add them to your shell profile (`~/.profile`). -```shell -alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" -alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk" -alias mariadb="docker run --rm -i --network=cratedb-demo docker.io/mariadb mariadb" -``` -::: -:::{tab-item} Windows PowerShell -To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). -```powershell -function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } -function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args } -function mariadb { docker run --rm -i --network=cratedb-demo docker.io/mariadb mariadb @args } -``` -::: -:::{tab-item} Windows Command ```shell -doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* -doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $* -doskey mariadb=docker run --rm -i --network=cratedb-demo docker.io/mariadb mariadb $* +docker compose up ``` -::: - -:::: -## Usage +## Submit data Write a few sample records to MariaDB. ::::{tab-set} -:::{tab-item} Linux and macOS +:::{tab-item} Linux, macOS, WSL ```shell -mariadb --protocol=tcp --host=mariadb --user=root --password=secret <init.sql < + postgres -c log_statement=all + environment: + POSTGRES_HOST_AUTH_METHOD: trust + ports: + - "6432:5432/tcp" + + ctk-ingest: + image: ghcr.io/crate/cratedb-toolkit-ingest:latest + profiles: ["cli"] diff --git a/docs/integrate/postgresql/usage.md b/docs/integrate/postgresql/usage.md index 1e7d1abe..b18dc42a 100644 --- a/docs/integrate/postgresql/usage.md +++ b/docs/integrate/postgresql/usage.md @@ -9,72 +9,40 @@ The data transfer is supported by the ## Prerequisites -Docker is used for running all components. This approach works consistently -across Linux, macOS, and Windows. Alternatively, you can use Podman. +Use Docker or Podman to run all components. This approach works consistently +across Linux, macOS, and Windows. -Create a shared network. -```shell -docker network create cratedb-demo -``` - -Start CrateDB. -```shell -docker run --rm --name=cratedb --network=cratedb-demo \ - --publish=4200:4200 --publish=5432:5432 \ - docker.io/crate -Cdiscovery.type=single-node -``` +### Files -Start PostgreSQL. -```shell -docker run --rm --name=postgresql --network=cratedb-demo \ - --publish=6432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" \ - docker.io/postgres postgres -c log_statement=all -``` -:::{note} -Because CrateDB is configured to listen on port `5432` with its PostgreSQL -interface, let's use a different port for PostgreSQL itself. -::: -:::{note} -Using `POSTGRES_HOST_AUTH_METHOD=trust` disables password checks. -Use it for local demos only. -::: +First, download and save all required files to your machine. +- {download}`compose.yaml` -Prepare shortcuts for the CrateDB shell, CrateDB Toolkit, and the PostgreSQL client -programs. +### Services -::::{tab-set} +Start services using Docker Compose or Podman Compose. +If you use Podman, replace `docker` with `podman` (or enable the podman‑docker +compatibility shim) and run `podman compose up`. -:::{tab-item} Linux and macOS -To make the settings persistent, add them to your shell profile (`~/.profile`). ```shell -alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" -alias ctk-ingest="docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk" -alias psql="docker run --rm -i --network=cratedb-demo docker.io/postgres psql" -``` -::: -:::{tab-item} Windows PowerShell -To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). -```powershell -function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } -function ctk-ingest { docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk @args } -function psql { docker run --rm -i --network=cratedb-demo docker.io/postgres psql @args } +docker compose up ``` + +:::{note} +CrateDB is configured to listen on port `5432`, +while PostgreSQL is configured to listen on port `6432`. ::: -:::{tab-item} Windows Command -```shell -doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* -doskey ctk-ingest=docker run --rm -i --network=cratedb-demo ghcr.io/crate/cratedb-toolkit-ingest ctk $* -doskey psql=docker run --rm -i --network=cratedb-demo docker.io/postgres psql $* -``` +:::{note} +PostgreSQL is configured using `POSTGRES_HOST_AUTH_METHOD=trust`. +This allows anonymous access for demonstration purposes only. +Do not expose it to untrusted networks. For production, configure +authentication/TLS. ::: -:::: - -## Usage +## Submit data Write a few sample records to PostgreSQL. ```shell -psql "postgresql://postgres:postgres@postgresql:5432/" < telegraf.conf -``` - -The database connection URL is a `pgx/v4` connection string. Configure -`table_create = true` to automatically let Telegraf create the metrics table -if it doesn't exist. -::::{tab-set} -:::{tab-item} Linux -```shell -sed -i 's!postgres://user:password@localhost/schema?sslmode=disable!postgres://crate@cratedb/doc?sslmode=disable!g' telegraf.conf -sed -i 's!# table_create = false!table_create = true!' telegraf.conf -``` -::: -:::{tab-item} macOS and BSD -```shell -sed -i '' 's!postgres://user:password@localhost/schema?sslmode=disable!postgres://crate@cratedb/doc?sslmode=disable!g' telegraf.conf -sed -i '' 's!# table_create = false!table_create = true!' telegraf.conf -``` -::: -:::{tab-item} Windows PowerShell -```powershell -(Get-Content telegraf.conf) -replace 'postgres://user:password@localhost/schema\?sslmode=disable','postgres://crate@cratedb/doc?sslmode=disable' | - ForEach-Object { $_ -replace '# table_create = false','table_create = true' } | - Set-Content telegraf.conf -``` -::: -:::: - - -Start Telegraf. -::::{tab-set} -:::{tab-item} Linux and macOS -```shell -docker run --name=telegraf --rm -it --network=cratedb-demo \ - --volume "$(pwd)"/telegraf.conf:/etc/telegraf/telegraf.conf \ - docker.io/telegraf -``` -::: -:::{tab-item} Windows PowerShell -```powershell -docker run --name=telegraf --rm -it --network=cratedb-demo ` - --volume "${PWD}\telegraf.conf:/etc/telegraf/telegraf.conf" ` - docker.io/telegraf -``` -::: -:::{tab-item} Windows Command -```shell -docker run --name=telegraf --rm -it --network=cratedb-demo ^ - --volume "%cd%\telegraf.conf:/etc/telegraf/telegraf.conf" ^ - docker.io/telegraf -``` -::: -:::: +of metrics. Because we just want to test things out, a single sensor is sufficient. +## Explore data After 10 seconds, which is the default output flush interval of Telegraf, the first metrics will appear in the `metrics` table in CrateDB. To adjust the value, navigate @@ -122,7 +40,19 @@ to `flush_interval = "10s"` in `telegraf.conf`. Inspect data stored in CrateDB. ```shell -crash --hosts cratedb -c "SELECT * FROM doc.metrics LIMIT 5;" +docker compose exec cratedb crash -c "SELECT * FROM doc.metrics LIMIT 5;" +``` +```psql ++----------------------+---------------+------+------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+ +| hash_id | timestamp | name | tags | fields | day | ++----------------------+---------------+------+------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+ +| -2149377133708231861 | 1759686560000 | cpu | {"cpu": "cpu5", "host": "02880df5c05e"} | {"usage_guest": 0, "usage_guest_nice": 0, "usage_idle": 98, "usage_iowait": 0, "usage_irq": 0, "usage_nice": 0, "usage_softirq": 0, "usage_steal": 0, "usage_system": 0, "usage_user": 0} | 1759622400000 | +| -4525358308278119095 | 1759686560000 | cpu | {"cpu": "cpu13", "host": "02880df5c05e"} | {"usage_guest": 0, "usage_guest_nice": 0, "usage_idle": 98, "usage_iowait": 0, "usage_irq": 0, "usage_nice": 0, "usage_softirq": 0, "usage_steal": 0, "usage_system": 0, "usage_user": 1} | 1759622400000 | +| 6120994000579036036 | 1759686560000 | cpu | {"cpu": "cpu14", "host": "02880df5c05e"} | {"usage_guest": 0, "usage_guest_nice": 0, "usage_idle": 99, "usage_iowait": 0, "usage_irq": 0, "usage_nice": 0, "usage_softirq": 0, "usage_steal": 0, "usage_system": 0, "usage_user": 0} | 1759622400000 | +| 869454636336868835 | 1759686560000 | cpu | {"cpu": "cpu1", "host": "02880df5c05e"} | {"usage_guest": 0, "usage_guest_nice": 0, "usage_idle": 99, "usage_iowait": 0, "usage_irq": 0, "usage_nice": 0, "usage_softirq": 0, "usage_steal": 0, "usage_system": 0, "usage_user": 0} | 1759622400000 | +| 722018796549427924 | 1759686560000 | cpu | {"cpu": "cpu8", "host": "02880df5c05e"} | {"usage_guest": 0, "usage_guest_nice": 0, "usage_idle": 97, "usage_iowait": 0, "usage_irq": 0, "usage_nice": 0, "usage_softirq": 0, "usage_steal": 0, "usage_system": 0, "usage_user": 1} | 1759622400000 | ++----------------------+---------------+------+------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+ +SELECT 5 rows in set (0.009 sec) ```