diff --git a/README.md b/README.md index affaa8ef..15756b42 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@
-## Intro +## Introduction _Scrolls_ is a tool for building and maintaining read-optimized collections of Cardano's on-chain entities. It crawls the history of the chain and aggregates all data to reflect the current state of affairs. Once the whole history has been processed, _Scrolls_ watches the tip of the chain to keep the collections up-to-date. @@ -16,6 +16,8 @@ Examples of collections are: "utxo by address", "chain parameters by epoch", "po > In other words, _Scrolls_ is just a map-reduce algorithm that aggregates the history of the chain into use-case-specific, key-value dictionaries. +Check our [documentation](https://txpipe.github.io/scrolls) for detailed information on how to start working with Scrolls. + :warning: this tool is under heavy development. Library API, configuration schema and storage structure may vary drastically. Several important features are still missing. Use at your own peril. ## Storage @@ -101,99 +103,6 @@ Scrolls is a pipeline that takes block data as input and outputs DB update comma - [ ] By Mint Policy / Asset - [ ] By Pool -## Testdrive - -In the `testdrive` folder you'll find a minimal example that uses docker-compose to spin up a local Redis instance and a Scrolls daemon. You'll need Docker and docker-compose installed in your local machine. Run the following commands to start it: - -```sh -cd testdrive -docker-compose up -``` - -You should see the logs of both _Redis_ and _Scrolls_ crawling the chain from a remote relay node. If you're familiar with Redis CLI, you can run the following commands to see the data being cached: - -```sh -redis:6379> KEYS * -1) "c1.addr1qx0w02a2ez32tzh2wveu80nyml9hd50yp0udly07u5crl6x57nfgdzya4axrl8mfx450sxpyzskkl95sx5l7hcfw59psvu6ysx" -2) "c1.addr1qx68j552ywp6engr2s9xt7aawgpmr526krzt4mmzc8qe7p8qwjaawywglaawe74mwu726w49e8e0l9mexcwjk4kqm2tq5lmpd8" -3) "c1.addr1q90z7ujdyyct0jhcncrpv5ypzwytd3p7t0wv93anthmzvadjcq6ss65vaupzmy59dxj43lchlra0l482rh0qrw474snsgnq3df" -4) "c1.addr1w8vg4e5xdpad2jt0z775rt0alwmku3my2dmw8ezd884zvtssrq6fg" -5) "c1.addr1q9tj3tdhaxqyph568h7efh6h0f078m2pxyd0xgzq47htwe3vep55nfane06hggrc2gvnpdj4gcf26kzhkd3fs874hzhszja3lh" -6) "c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz" -redis:6379> SMEMBERS c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz -1) "2548228522837ea580bc55a3e6a09479deca499b5e7f3c08602a1f3191a178e7:20" -2) "04086c503512833c7a0c11fc85f7d0f0422db9d14b31275b3d4327c40c6fd73b:25" -redis:6379> -``` - -Once you're done with the testdive, you can clean your environment by running: - -```sh -docker-compose down -``` - -## Installing - -We currently provide the following ways to install _Scrolls_: - -- Using one of the pre-compiled binaries shared via [Github Releases](https://github.com/txpipe/scrolls/releases) -- Using the Docker image shared via [Github Packages](https://github.com/txpipe/scrolls/pkgs/container/scrolls) -- By compiling from source code using the instructions provided in this README. - - -## Configuration - -This is an example configuration file: - -```toml -# get data from a relay node -[source] -type = "N2N" -address = "relays-new.cardano-mainnet.iohk.io:3001" - -# You can optionally enable enrichment (local db with transactions), this is needed for some reducers -[enrich] -type = "Sled" -db_path = "/opt/scrolls/sled_db" - -# enable the "UTXO by Address" collection -[[reducers]] -type = "UtxoByAddress" -# you can optionally prefix the keys in the collection -key_prefix = "c1" -# you can optionally only process UTXO from a set of predetermined addresses -filter = ["addr1qy8jecz3nal788f8t2zy6vj2l9ply3trpnkn2xuvv5rgu4m7y853av2nt8wc33agu3kuakvg0kaee0tfqhgelh2eeyyqgxmxw3"] - -# enable the "Point by Tx" collection -[[reducers]] -type = "PointByTx" -key_prefix = "c2" - -# store the collections in a local Redis -[storage] -type = "Redis" -connection_params = "redis://127.0.0.1:6379" - -# start reading from an arbitrary point in the chain -[intersect] -type = "Point" -value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"] - -# let Scrolls know that we're working with mainnet -[chain] -type = "Mainnet" -``` - -## Compiling from Source - -To compile from source, you'll need to have the Rust toolchain available in your development box. Execute the following command to clone and build the project: - -```sh -git clone https://github.com/txpipe/scrolls.git -cd scrolls -cargo build -``` - ## FAQ ### Don't we have tools for this already? @@ -211,24 +120,6 @@ Yes, we do. We have excellent tools such as: [Kupo](https://github.com/CardanoSo There's some overlap between _Oura_ and _Scrolls_. Both tools read on-chain data and output some data results. The main difference is that Oura is meant to **_react_** to events, to watch the chain and actuate upon certain patterns. In contrast, _Scrolls_ is meant to provide a snapshot of the current state of the chain by aggregating the whole history. They were built to work well together. For example, let's say that you're building an app that uses Oura to process transaction data, you could then integrate _Scrolls_ as a way to lookup the source address of the transaction's input. - -### How do I read the data using Python? - -Assuming you're using Redis as a storage backend (only one available ATM), we recommend using [redis-py](https://github.com/redis/redis-py) package to talk directly to the Redis instance. This is a very simple code snippet to query a the UTXOs by address. - -```python ->>> import redis ->>> r = redis.Redis(host='localhost', port=6379, db=0) ->>> r.smembers("c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz") -{b'2548228522837ea580bc55a3e6a09479deca499b5e7f3c08602a1f3191a178e7:20', b'04086c503512833c7a0c11fc85f7d0f0422db9d14b31275b3d4327c40c6fd73b:25'} -``` - - The Redis operation being used is `smembers` which return the list of members of a set stored under a particular key. In this case, we query by the value `c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz`, where `c1` is the key prefix specified in the config for our particular collection and `addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz` is the address we're interested in querying. The response from redis is the list of UTXOs (in the format `{tx-hash}:{output-index}`) that are associated with that particular address. - -### How do I read the data using NodeJS? - -TODO - ### What is "swarm mode"? Swarm mode is a way to speed up the process of rebuilding collection from scratch by splitting the tasks into concurrent instances of the _Scrolls_ daemon by partitioning the history of the chain into smaller fragments. diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index e0ff9f90..069101c5 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -1,3 +1,26 @@ # Summary - [Introduction](./introduction.md) +- [Installation](./installation/README.md) + - [Binary Release](./installation/binary_release.md) + - [From Source](./installation/from_source.md) + - [Docker](./installation/docker.md) +- [Usage](./usage/README.md) +- [Configuration](./configuration/README.md) + - [Sources](./configuration/sources.md) + - [Reducers](./configuration/reducers/README.md) + - [Predicates](./configuration/reducers/predicates.md) + - [Storage](./configuration/storage.md) + - [Enrich](./configuration/enrich.md) + - [Intersect](./configuration/intersect.md) + - [Chain](./configuration/chain.md) + - [Policy](./configuration/policy.md) +- [Advanced Features](./advanced/README.md) + - [Swarm Mode](./advanced/swarm_mode.md) +- [Troubleshooting](./troubleshooting/README.md) +- [Guides](./guides/README.md) + - [Testdrive](./guides/testdrive.md) + - [NodeJS Client](./guides/nodejs.md) + - [Python Client](./guides/python.md) + - [Redis-cli](./guides/redis.md) + diff --git a/book/src/advanced/README.md b/book/src/advanced/README.md new file mode 100644 index 00000000..7c935bda --- /dev/null +++ b/book/src/advanced/README.md @@ -0,0 +1 @@ +# Advanced features diff --git a/book/src/advanced/swarm_mode.md b/book/src/advanced/swarm_mode.md new file mode 100644 index 00000000..4ea185a4 --- /dev/null +++ b/book/src/advanced/swarm_mode.md @@ -0,0 +1,4 @@ +# Swarm mode + +Swarm mode is a way to speed up the process of rebuilding collection from scratch by splitting the tasks into concurrent instances of the Scrolls daemon by partitioning the history of the chain into smaller fragments. + diff --git a/book/src/configuration/README.md b/book/src/configuration/README.md new file mode 100644 index 00000000..9fded78f --- /dev/null +++ b/book/src/configuration/README.md @@ -0,0 +1,48 @@ +# Configuration +For the purpose of testing out Scrolls you can use the provided configuration located in `testdrive/simple/daemon.toml`. See below for another example with explanations and check the following sections of this book to understand in detail each section of the configuration file. + +## Format +Scrolls daemon supports `.toml` and `.json` configuration files. Unlike json, toml supports comments which are very handy to turn declarations on and off, specially during early stages of development, debugging, learning, etc. On the other hand, deeply nested filters can be difficult to understand using toml syntax, so the user can choose to declare the whole configuration with json, or instead to rely on tools like [toml2json](https://github.com/woodruffw/toml2json) and [remarshal](https://github.com/remarshal-project/remarshal) to translate small chunks of json (such as complex deeply nested filters) to be used in toml configuration files. + +When working with toml configuration files, sometimes it also helps to translate the whole configuration to json, and use [jq](https://stedolan.github.io/jq/)/[bat](https://github.com/sharkdp/bat) to make the json human friendly. This often helps to understand the structure of the filters. Example: `toml2json ./configuration.toml | jq | bat -l json` + +## Configuration Example +```toml +# get data from a relay node +[source] +type = "N2N" +address = "relays-new.cardano-mainnet.iohk.io:3001" + +# You can optionally enable enrichment (local db with transactions), this is needed for some reducers +[enrich] +type = "Sled" +db_path = "/opt/scrolls/sled_db" + +# enable the "UTXO by Address" collection +[[reducers]] +type = "UtxoByAddress" +# you can optionally prefix the keys in the collection +key_prefix = "c1" +# you can optionally only process UTXO from a set of predetermined addresses +filter = ["addr1qy8jecz3nal788f8t2zy6vj2l9ply3trpnkn2xuvv5rgu4m7y853av2nt8wc33agu3kuakvg0kaee0tfqhgelh2eeyyqgxmxw3"] + +# enable the "Point by Tx" collection +[[reducers]] +type = "PointByTx" +key_prefix = "c2" + +# store the collections in a local Redis +[storage] +type = "Redis" +connection_params = "redis://127.0.0.1:6379" + +# start reading from an arbitrary point in the chain +[intersect] +type = "Point" +value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"] + +# let Scrolls know that we're working with mainnet +[chain] +type = "Mainnet" +``` + diff --git a/book/src/configuration/chain.md b/book/src/configuration/chain.md new file mode 100644 index 00000000..b4f9a659 --- /dev/null +++ b/book/src/configuration/chain.md @@ -0,0 +1,50 @@ +# Chain + +Specify which network to fetch data from. + +## Fields +- type: `"Mainnet" | "Testnet" | "PreProd" | "Preview" | "Custom"` +- magic (*): `u64`, +- byron_epoch_length (*): `u32`, +- byron_slot_length (*): `u32`, +- byron_known_slot (*): `u64`, +- byron_known_hash (*): `String`, +- byron_known_time (*): `u64`, +- shelley_epoch_length (*): `u32`, +- shelley_slot_length (*): `u32`, +- shelley_known_slot (*): `u64`, +- shelley_known_hash (*): `String`, +- shelley_known_time (*): `u64`, +- address_network_id (*): `u8`, +- adahandle_policy (*): `String`, + + +(*) Use only with `type = "Custom"` + +## Examples + +Using mainnet +``` toml +[chain] +type = "Mainnet" +``` + +Using custom values (mainnet): +``` toml +[chain] +type = "Custom" +magic = 764824073 +byron_epoch_length = 432000 +byron_slot_length = 20 +byron_known_slot = 0 +byron_known_time = 1506203091 +byron_known_hash = "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f" +shelley_epoch_length = 432000 +shelley_slot_length = 1 +shelley_known_slot = 4492800 +shelley_known_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" +shelley_known_time = 1596059091 +address_network_id = 1 +adahandle_policy = "f0ff48bbb7bbe9d59a40f1ce90e9e9d0ff5002ec48f232b49ca0fb9a" + +``` diff --git a/book/src/configuration/enrich.md b/book/src/configuration/enrich.md new file mode 100644 index 00000000..346f980c --- /dev/null +++ b/book/src/configuration/enrich.md @@ -0,0 +1,16 @@ +# Enrich +Store utxo information in a local DB, this is needed for some reducers to work. Currently, only [Sled](https://github.com/spacejam/sled) databases are supported. + +## Fields +- type: `"Sled" | "Skip"` +- db_path (*): `String` + +(*) Use only with `type = "Sled"` + +## Example + +``` toml +[enrich] +type = "Sled" +db_path = "/opt/scrolls/sled_db" +``` diff --git a/book/src/configuration/intersect.md b/book/src/configuration/intersect.md new file mode 100644 index 00000000..51dbdb7b --- /dev/null +++ b/book/src/configuration/intersect.md @@ -0,0 +1,34 @@ +# Intersect + +Scrolls provides 4 different strategies for finding the intersection point within the chain sync process. + +- `Origin`: Scrolls will start reading from the beginning of the chain. +- `Tip`: Scrolls will start reading from the current tip of the chain. +- `Point`: Scrolls will start reading from a particular point (slot, hash) in the chain. If the point is not found, the process will be terminated with a non-zero exit code. +- `Fallbacks`: Scrolls will start reading the first valid point within a set of alternative positions. If point is not valid, the process will fallback into the next available point in the list of options. If none of the points are valid, the process will be terminated with a non-zero exit code. + + +## Fields +- type: `"Tip" | "Origin" | "Point" | "Fallbacks"` +- value (*): `(u64, String) | Vec<(u64, String)>` + +(*) Use value of type `(u64, String)` with `type = "Point"` and value of type `Vec<(u64, String)>` with `type = "Fallbacks"` + +## Examples + +Using **Point**: +``` toml +[intersect] +type = "Point" +value = [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"] +``` + +Using **Fallbacks**: +``` toml +[intersect] +type = "Fallbacks" +value = [ + [12345678, "this_is_not_a_valid_hash_ff1b93cdfd997d4ea93e7d930908aa5905d788f"], + [57867490, "c491c5006192de2c55a95fb3544f60b96bd1665accaf2dfa2ab12fc7191f016b"] + ] +``` diff --git a/book/src/configuration/policy.md b/book/src/configuration/policy.md new file mode 100644 index 00000000..7b81d06b --- /dev/null +++ b/book/src/configuration/policy.md @@ -0,0 +1,17 @@ +# Policy + +## Fields +- missing_data: `"Skip" | "Warn" | "Default"` +- cbor_errors: `"Skip" | "Warn" | "Default"` +- ledger_errors: `"Skip" | "Warn" | "Default"` +- any_error: `"Skip" | "Warn" | "Default"` + + +## Example + +``` toml +[policy] +cbor_errors = "Skip" +missing_data = "Warn" +``` + diff --git a/book/src/configuration/reducers/README.md b/book/src/configuration/reducers/README.md new file mode 100644 index 00000000..c0217ef6 --- /dev/null +++ b/book/src/configuration/reducers/README.md @@ -0,0 +1,256 @@ +# Reducers + + +- [address_by_asset](#address_by_asset) +- [address_by_txo](#address_by_txo) +- [addresses_by_stake](#addresses_by_stake) +- [asset_holders_by_asset_id](#asset_holders_by_asset_id) +- [balance_by_address](#balance_by_address) +- [block_header_by_hash](#block_header_by_hash) +- [last_block_parameters](#last_block_parameters) +- [point_by_tx](#point_by_tx) +- [pool_by_stake](#pool_by_stake) +- [supply_by_asset](#supply_by_asset) +- [tx_by_hash](#tx_by_hash) +- [tx_count_by_address](#tx_count_by_address) +- [tx_count_by_native_token_policy_id](#tx_count_by_native_token_policy_id) +- [utxo_by_address](#utxo_by_address) +- [utxo_by_stake](#utxo_by_stake) +- [utxos_by_asset](#utxos_by_asset) + + +## address_by_asset + +### Config + +- key_prefix: `Option` +- filter: `Option>` +- policy_id_hex: `String` +- convert_to_ascii(*): `Option` + +(*) default is true. + +### Example + +### Output Format + +
+
+
+ +## address_by_txo + +### Config + +- key_prefix: `Option` +- filter (*): `Option` + +(*) See: [Predicates](./predicates.md) + + +### Example +### Output Format + +
+
+
+ +## addresses_by_stake + +### Config + +- key_prefix: `Option` +- filter: `Option>` + +### Example +### Output Format + +
+
+
+ +## asset_holders_by_asset_id + +### Config + +- key_prefix: `Option` +- filter (*): `Option` +- aggr_by (**): `Option` + +- policy_ids_hex: `Option>` + +(*) See: [Predicates](./predicates.md) + +(**) Policies to match. If specified only those policy ids as hex will be taken into account, if not all policy ids will be indexed. + +### Example +### Output Format + +
+
+
+ +## balance_by_address + +### Config +- key_prefix: `Option` +- filter (*): `Option` + +(*) See: [Predicates](./predicates.md) + +### Example +### Output Format + +
+
+
+ +## block_header_by_hash + +### Config +- key_prefix: `Option` +- filter (*): `Option` + +(*) See: [Predicates](./predicates.md) + +### Example +### Output Format + +
+
+
+ +## last_block_parameters + +### Config +- key_prefix: `Option` + +### Example +### Output Format + +
+
+
+ +## point_by_tx + +### Config +- key_prefix: `Option` + +### Example +### Output Format + +
+
+
+ +## pool_by_stake + +### Config +- key_prefix: `Option` + +### Example +### Output Format + +
+
+
+ +## supply_by_asset + +### Config +- key_prefix: `Option` +- policy_ids_hex: `Option>` + +### Example +### Output Format + +
+
+
+ +## tx_by_hash + +### Config +- key_prefix: `Option` +- filter (*): `Option` +- projection: `"Cbor" | "Json"` + +(*) See: [Predicates](./predicates.md) + +### Example + +``` toml +[[reducers]] +type: "TxByHash" +key_prefix: "c1" +``` + +### Output Format + +
+
+
+ +## tx_count_by_address + +### Config +- key_prefix: `Option` +- filter (*): `Option` + +(*) See: [Predicates](./predicates.md) + +### Example +### Output Format + +
+
+
+ +## tx_count_by_native_token_policy_id + +### Config +- key_prefix: `Option` +- aggr_by: `Option` + +### Example +### Output Format + +
+
+
+ +## utxo_by_address + +### Config +- key_prefix: `Option` +- filter: `Option>` + +### Example +### Output Format + +
+
+
+ +## utxo_by_stake + +### Config +- key_prefix: `Option` +- filter: `Option>` + +### Example +### Output Format + +
+
+
+ +## utxos_by_asset + +### Config +- key_prefix: `Option` +- policy_ids_hex: `Option>` + +### Example +### Output Format diff --git a/book/src/configuration/reducers/predicates.md b/book/src/configuration/reducers/predicates.md new file mode 100644 index 00000000..0d6e5f42 --- /dev/null +++ b/book/src/configuration/reducers/predicates.md @@ -0,0 +1,401 @@ +# Predicates + +Following `Predicate`s are available to be used as filters by some reducers. + +- [all_of](#all_of-vecpredicate) `Vec` +- [any_of](#any_of-vecpredicate) `Vec` +- [not](#not-predicate) `Predicate` +- [block](#block-blockpattern) `BlockPattern` +- [transaction](#transaction-transactionpattern) `TransactionPattern` +- [input_address](#input_address-addresspattern) `AddressPattern` +- [output_address](#output_address-addresspattern) `AddressPattern` +- [withdrawal_address](#withdrawal_address-addresspattern) `AddressPattern` +- [collateral_address](#collateral_address-addresspattern) `AddressPattern` +- [address](#address-addresspattern) `AddressPattern` + + +##### BlockPattern: +- slot_before: `Option` +- slot_after: `Option` + +##### TransactionPattern: +- is_valid: `Option` + +##### AddressPattern: +- exact_hex: `Option` +- exact_bech32: `Option` +- payment_hex: `Option` +- payment_bech32: `Option` +- stake_hex: `Option` +- stake_bech32: `Option` +- is_script: `Option` + +
+
+
+ +## `all_of (Vec)` + This predicate will yield true when _all_ of the predicates in the argument yields true. + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[[reducers.filter.all_of]] +[reducers.filter.all_of.output_address] +exact = "
" + +[[reducers.filter.all_of]] +[reducers.filter.all_of.input_address] +exact = "
" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "all_of": [ + { + "output_address": { + "exact": "
" + } + }, + { + "input_address": { + "exact": "
" + } + } + ] + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ + +## `any_of (Vec)` + This predicate will yield true when _any_ of the predicates in the argument yields true. + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[[reducers.filter.any_of]] +[reducers.filter.any_of.output_address] +exact = "
" + +[[reducers.filter.any_of]] +[reducers.filter.any_of.input_address] +exact = "
" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "any_of": [ + { + "output_address": { + "exact": "
" + } + }, + { + "input_address": { + "exact": "
" + } + } + ] + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## not (Predicate) + This predicate will yield true when the predicate in the arguments yields false. + + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[[reducers.filter.not]] +[reducers.filter.not.output_address] +exact = "" + +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "type": "AddressByTxo", + "filter": { + "not": { + "output_address": { + "exact": "" + } + } + } + } + ] +} +``` + +
+
+
+ +## `block (BlockPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddresByTxo" + +[[reducers.filter.block]] +slot_before = 83548786 +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "type": "AddresByTxo", + "filter": { + "block": [ + { + "slot_before": 83548786 + } + ] + } + } + ] +} +``` + +
+
+
+ +## `transaction (TransactionPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.transaction] +is_valid = false + +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "transaction": { + "is_valid": false + } + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## `input_address (AddressPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.input_address] +exact_hex = "" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "input_address": { + "exact_hex": "" + } + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## `output_address (AddressPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.output_address] +exact_hex = "" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "output_address": { + "exact_hex": "" + } + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## `withdrawal_address (AddressPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.withdrawal_address] +exact_hex = "" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "withdrawal_address": { + "exact_hex": "" + } + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## `collateral_address (AddressPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.collateral_address] +exact_hex = "" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "collateral_address": { + "exact_hex": "" + } + }, + "type": "AddressByTxo" + } + ] +} +``` + +
+
+
+ +## `address (AddressPattern)` + +#### Example (toml) + +```toml +[[reducers]] +type = "AddressByTxo" + +[reducers.filter.address] +exact_hex = "" +``` + +#### Example (json) + +```json +{ + "reducers": [ + { + "filter": { + "address": { + "exact_hex": "" + } + }, + "type": "AddressByTxo" + } + ] +} +``` diff --git a/book/src/configuration/sources.md b/book/src/configuration/sources.md new file mode 100644 index 00000000..b34718c2 --- /dev/null +++ b/book/src/configuration/sources.md @@ -0,0 +1,31 @@ +# Sources + +Sources represent the "origin" of the events processed by Scrolls. Any compatible source is responsible for feeding blockchain data into crolls's pipeline for further processing. This section describes the currently available sources included as part the main Scrolls codebase. + +The **Node-to-Node (N2N)** source uses Ouroboros mini-protocols to connect to a local or remote Cardano node through a tcp socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "headers only" and the BlockFetch mini-protocol for retrieval of the actual block payload. + +The **Node-to-Client (N2C)** source uses Ouroboros mini-protocols to connect to a local Cardano node through a unix socket bearer and fetches block data using the ChainSync mini-protocol instantiated to "full blocks". + +## Fields + +- type: `"N2N" | "N2C"` +- address (*): `String` +- path (**): `String` +- min_depth: `"Option"` + +(*) Use only with `type = "N2N"` + +(**) Use only with `type = "N2C` + +## Examples +``` toml +[source] +type = "N2N" +address = "relays-new.cardano-mainnet.iohk.io:3001" +``` + +``` toml +[source] +type = "N2C" +path = "/path/to/node.socket" +``` diff --git a/book/src/configuration/storage.md b/book/src/configuration/storage.md new file mode 100644 index 00000000..82a0a5e4 --- /dev/null +++ b/book/src/configuration/storage.md @@ -0,0 +1,23 @@ +# Storage + +Storage backends are "pluggable", any key-value storage mechanism is a potential candidate. Our backend of preference is Redis (and TBH, the only one implemented so far). It provides a very high "read" throughput, it can be shared across the network by multiple clients and can be used in cluster-mode for horizontal scaling. + +In case you don't have a redis instance running in your system, you might want to look at our [Testdrive](../guides/testdrive.md) guide for a minimal docker example providing a local redis instance. You can also look at the official Redis [Installation guide](https://redis.io/docs/getting-started/installation/). + +See our [Redis-cli basics](../guides/redis.md) guide to get started playing with Redis from the command line. + +### Fields + +- type: `"Redis" | "Skip"` +- connection_params (*): `String` +- cursor_key (*): `Option` + +(*) Use only with `type = "Redis"` + +#### Example + +``` toml +[storage] +type = "Redis" +connection_params = "redis://127.0.0.1:6379" +``` diff --git a/book/src/guides/README.md b/book/src/guides/README.md new file mode 100644 index 00000000..c2fe4136 --- /dev/null +++ b/book/src/guides/README.md @@ -0,0 +1,3 @@ +# Guides + +Quick informal guides to get starting using Scrolls with external tools. diff --git a/book/src/guides/nodejs.md b/book/src/guides/nodejs.md new file mode 100644 index 00000000..cd574bc4 --- /dev/null +++ b/book/src/guides/nodejs.md @@ -0,0 +1 @@ +# NodeJS Client diff --git a/book/src/guides/python.md b/book/src/guides/python.md new file mode 100644 index 00000000..0fb444f8 --- /dev/null +++ b/book/src/guides/python.md @@ -0,0 +1,13 @@ +# Python Client + +Assuming you're using Redis as a storage backend (only one available ATM), we recommend using [redis-py](https://github.com/redis/redis-py) package to talk directly to the Redis instance. This is a very simple code snippet to query a the UTXOs by address. + +```python +>>> import redis +>>> r = redis.Redis(host='localhost', port=6379, db=0) +>>> r.smembers("c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz") +{b'2548228522837ea580bc55a3e6a09479deca499b5e7f3c08602a1f3191a178e7:20', b'04086c503512833c7a0c11fc85f7d0f0422db9d14b31275b3d4327c40c6fd73b:25'} +``` + + The Redis operation being used is `smembers` which return the list of members of a set stored under a particular key. In this case, we query by the value `c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz`, where `c1` is the key prefix specified in the config for our particular collection and `addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz` is the address we're interested in querying. The response from redis is the list of UTXOs (in the format `{tx-hash}:{output-index}`) that are associated with that particular address. + diff --git a/book/src/guides/redis.md b/book/src/guides/redis.md new file mode 100644 index 00000000..181a3711 --- /dev/null +++ b/book/src/guides/redis.md @@ -0,0 +1,10 @@ +# Redis-cli + +If you are not familiar with Redis you might want to play with the [redis-cli](https://redis.io/docs/ui/cli/) command line application before start coding any serious application. This guide gives you the very basic `redis-cli` commands needed to fetch information from redis databases. + +- Get all keys: `KEYS *` +- Get type of key: `TYPE` +- Get set members: `SMEMBERS` +- Get string: `GET` +- Clear database: `FLUSHDB` +- Clear all databases: `FLUSHALL` diff --git a/book/src/guides/testdrive.md b/book/src/guides/testdrive.md new file mode 100644 index 00000000..fbf3c1aa --- /dev/null +++ b/book/src/guides/testdrive.md @@ -0,0 +1,31 @@ +# Testdrive + +In the `testdrive` folder you'll find a minimal example that uses docker-compose to spin up a local Redis instance and a Scrolls daemon. You'll need Docker and docker-compose installed in your local machine. Run the following commands to start it: + +```sh +cd testdrive +docker-compose up +``` + +You should see the logs of both _Redis_ and _Scrolls_ crawling the chain from a remote relay node. If you're familiar with Redis CLI, you can run the following commands to see the data being cached: + +```sh +redis:6379> KEYS * +1) "c1.addr1qx0w02a2ez32tzh2wveu80nyml9hd50yp0udly07u5crl6x57nfgdzya4axrl8mfx450sxpyzskkl95sx5l7hcfw59psvu6ysx" +2) "c1.addr1qx68j552ywp6engr2s9xt7aawgpmr526krzt4mmzc8qe7p8qwjaawywglaawe74mwu726w49e8e0l9mexcwjk4kqm2tq5lmpd8" +3) "c1.addr1q90z7ujdyyct0jhcncrpv5ypzwytd3p7t0wv93anthmzvadjcq6ss65vaupzmy59dxj43lchlra0l482rh0qrw474snsgnq3df" +4) "c1.addr1w8vg4e5xdpad2jt0z775rt0alwmku3my2dmw8ezd884zvtssrq6fg" +5) "c1.addr1q9tj3tdhaxqyph568h7efh6h0f078m2pxyd0xgzq47htwe3vep55nfane06hggrc2gvnpdj4gcf26kzhkd3fs874hzhszja3lh" +6) "c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz" +redis:6379> SMEMBERS c1.addr1w8tqqyccvj7402zns2tea78d42etw520fzvf22zmyasjdtsv3e5rz +1) "2548228522837ea580bc55a3e6a09479deca499b5e7f3c08602a1f3191a178e7:20" +2) "04086c503512833c7a0c11fc85f7d0f0422db9d14b31275b3d4327c40c6fd73b:25" +redis:6379> +``` + +Once you're done with the testdive, you can clean your environment by running: + +```sh +docker-compose down +``` + diff --git a/book/src/installation/README.md b/book/src/installation/README.md new file mode 100644 index 00000000..18697015 --- /dev/null +++ b/book/src/installation/README.md @@ -0,0 +1,10 @@ +# Installation + +Depending on your needs, Scrolls provides different installation options: + +- [Binary Release](binary_release.md): to use one of our pre-compiled binary releases for the supported platforms. +- [From Source](from_source.md): to compile a binary from source code using Rust's toolchain +- [Docker](docker.md): to run the tool from a pre-built docker image + + +You will also need a data storage backend installed in your system. Currently we support Redis, see our [Testdrive](../guides/testdrive.md) guide for a minimal docker image providing a Redis instance. diff --git a/book/src/installation/binary_release.md b/book/src/installation/binary_release.md new file mode 100644 index 00000000..5b4c8c62 --- /dev/null +++ b/book/src/installation/binary_release.md @@ -0,0 +1,7 @@ +# Binary release + +Installation using the pre-compiled binaries shared via [Github Releases](https://github.com/txpipe/scrolls/releases) + +# Linux + +Binaries built with musl are statically linked. diff --git a/book/src/installation/docker.md b/book/src/installation/docker.md new file mode 100644 index 00000000..52d0a40d --- /dev/null +++ b/book/src/installation/docker.md @@ -0,0 +1,11 @@ +# Docker + +Installation using the Docker image. Use the following command: + +``` +docker pull ghcr.io/txpipe/scrolls:v0.5.0 +``` + +Check [Github Packages](https://github.com/txpipe/scrolls/pkgs/container/scrolls) for newer versions. + +Check the [Testdrive](../guides/testdrive.md) guide for a minimal example that uses docker-compose to spin up a local Redis instance and a Scrolls daemon. diff --git a/book/src/installation/from_source.md b/book/src/installation/from_source.md new file mode 100644 index 00000000..c81061a7 --- /dev/null +++ b/book/src/installation/from_source.md @@ -0,0 +1,25 @@ +# From Source + + +To compile from source, you'll need to have the Rust toolchain available in your development box. Execute the following commands to clone and build the project: + +```sh +git clone https://github.com/txpipe/scrolls.git +cd scrolls +cargo build +``` + +Tell cargo to use git to fetch dependecies if it fails with `no authentication available` error: +``` +CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build +``` + +Execute scrolls with: + +``` +./target/debug/scrolls +``` + + + +TODO: `cargo install --path .` diff --git a/book/src/introduction.md b/book/src/introduction.md index 4f3b828c..4aeb6926 100644 --- a/book/src/introduction.md +++ b/book/src/introduction.md @@ -3,3 +3,6 @@ ## Motivation Since blockchain is a sequence of transactions it is far from trivial to get a fast read only access to data. Scrolls is a tool for building and maintaining read-optimized collections of Cardano's on-chain entities. It crawls the history of the chain and aggregates all data to reflect the current state of affairs. Once the whole history has been processed, Scrolls watches the tip of the chain to keep the collections up-to-date. + +## Under the hood +All the heavy lifting required to communicate with the Cardano node is done by the [Pallas](https://github.com/txpipe/pallas) library, which provides an implementation of the Ouroboros multiplexer and a few of the required mini-protocol state-machines (ChainSync and LocalState in particular). diff --git a/book/src/troubleshooting/README.md b/book/src/troubleshooting/README.md new file mode 100644 index 00000000..76827252 --- /dev/null +++ b/book/src/troubleshooting/README.md @@ -0,0 +1,25 @@ +# Troubleshooting + +## Error: Missing utxo + +``` +STAGE: reducers, WORK, PANIC: missing utxo: # +``` + +When processing a particular transaction, some reducers could need inputs to be cached in a local database, this is called [enrichment](../configuration/enrich.md). To avoid facing this error, you may need to run Scrolls from origin or some old [intersection point](../configuration/intersect.md), to be sure you have every needed tx input cached in the local db. + +Alternatively, you may want to skip this error with: + +``` + [policy] +missing_data = "Skip" +``` + +## Error: chain-sync intersect not found + +``` +STAGE: n2n, BOOTSTRAP, PANIC: chain-sync intersect not found +``` + + +Stored `_cursor` value doesn't point to a valid pair of `"absolute_slot,block_hash"` diff --git a/book/src/usage/README.md b/book/src/usage/README.md new file mode 100644 index 00000000..1b52724a --- /dev/null +++ b/book/src/usage/README.md @@ -0,0 +1,11 @@ +# Usage + +Once you have Scrolls and Redis installed in your system (see [Installation](../installation/index.md)) you can start the daemon with the following command: + + +``` bash +/path/to/scrolls daemon --config /path/to/config.toml +``` + +Check the [Configuration](../configuration/index.md) section of the manual for further information. If you have no experience using Redis, you can follow the [Redis-cli guide](../guides/redis.md) to learn the basic commands needed to query data using the command line application provided by Redis. For any real application you would need to use a client library in your language of preference. See our [Python](../guides/python.md) and [NodeJS](../guides/nodejs.md) guides. +