Advocate

Advocate - an Agent for Data Verification, On-Chaining and Anchoring in the Trustless Environments

Advocate is a tool and APIs to enable Kubernetes-based applications to verify service interactions and anchor data in the TEADAL environment. It functions as a trust provider for interactions with the TEADAL FDPs, facilitating the collection of verifiable evidence utilizing infrastructure observability tools such as Jaeger, Istio, and ENS.

Table of Contents

• Prerequisites
• Installation
• Deployment
• Configuration
• Operation
• Further Documentation
• Data Model
• Authors
• License

Prerequisites

• Python 3.9 or higher
• Pipenv
• docker
• vscode and the python extension or IntelliJ PyCharm

Installation

git clone <this repo>
cd advocate_pie

If you want to use Rego based Aggregation in your Advocate instance, you need to install the Rego wheel for your machine.

# MacOS
pipenv install packages/regorus/regorus-0.2.2-cp39-cp39-macosx_11_0_arm64.whl
# Linux
pipenv install packages/regorus/regorus-0.2.2-cp39-cp39-manylinux_2_34_x86_64.whl

If your are running on other/custom setups you will need to build the wheel yourself and install it. Follow the instruction on Github microsoft/regorus

More information about Aggregations and definition of policies can be found in the Rego Aggregator README.

Install dependencies:

pipenv install
pipenv install --dev

Setup

Before starting Advocate you need to generate the certificates for the Advocate instance.

1. Find the IP Address of you local machine
2. Add it to the Following script at <YOUR IP ADDRESS> and run it in the terminal:

ADVOCATE_SELF_ADDRESSES=https://advocate.trust-plane.svc.cluster.local:8000,https://130.149.158.50:8000,https://127.0.0.1:8000,https://localhost:8000/,https://<YOUR IP ADDRESS>:8080 ./docker/gen-env-certs.sh

4. Add your local IP Address to the .secrets.toml file as SELF_ADDRESSES and PUBLIC_ADDRESS with the following format:

SELF_ADDRESSES = "https://<Your IP ADDRESS>:8080"
PUBLIC_ADDRESS = "https://<Your IP ADDRESS>:8080"

To make us of Advocate's full range of capabilities, the federation contract must be deployed first. Look at the Federation Contract section for more information on how to deploy the federation contract.

Deployment

To start Advocate, you need to provide an environment that has access to an IPFS node and a Jaeger instance and an Ethereum network, e.g., ganache. For development purposes, you can use the following docker-compose file to start an IPFS node and a Jaeger instance.

cd e2e
./run_test_env.sh

Advocate uses a configuration file, which you need to define. See the settings.toml file for an example. The Configuration Reference lists the available configuration options. Moreover, to use the local test environment you will need to define some secret keys in the .secrets.toml file.

WALLET_PRIVATEKEY_FILE = "0x22aabb811efca4e6f4748bd18a46b502fa85549df9fa07da649c0a148d7d5530" 

For local execution and debugging of Advocate, two configurations are available:

1. Advocate (Script Execution)

The standard debug configuration

IDE	Configuration Details
VS Code	Use the Advocate configuration defined in .vscode/launch.json.
IntelliJ	Use the Advocate configuration defined in .run/Advocate.run.xml.

Test if Advocate is running

curl -k https://localhost:8080/api/v1/health

2. Advocate (Gunicorn Execution)

This uses the Gunicorn WSGI server

IDE	Configuration
VS Code	Use the Advocate (Gunicorn) configuration defined in .vscode/launch.json
IntelliJ	Use the Advocate (Gunicorn) configuration defined in .run/Advocate (Gunicorn).run.xml

Test if Advocate is running

curl -k https://localhost:8080/api/v1/health

Configuration

The following table lists the available configuration options for Advocate. They can be set to the settings.toml or using environment variables with the ADVOCATE_ prefix.

Key	Description	Example
JAEGER_QUERY_ADDRESS	Address for Jaeger Query API at port 16685	"localhost:16685"
IPFS_CONNECTION	IPFS connection URL	"http://localhost:5001/api/v0/"
WALLET_PRIVATEKEY	The Ethereum wallet private key of the Advocate operator. Can be a hex string of the key or a path to a file. Will attempt to read or ask for a password for decyption.	"~/.wallte"
ETH_RPC_ADDRESS	The address of your blockchain network	"http://localhost:7545"
DOCUMENT_STORE_CONTRACT_ADDRESS	The address of the document stores smart contracts on the blockchain. If empty, we deploy it.	"0x0"
DOCUMENT_STORE_CONTRACT_ABI_PATH	The path to the document store contract ABI file.	"advocate/core/ethereum/contracts/DocumentStore.json"
DOCUMENT_STORE_NAME	The name of the document store contract.	"TUB DS"
VM_KEY	A unique key of the VM running this Advocate instance	"1231das4"
PUBLIC_ADDRESS	the address this Advocate is reachable at publicly	"advocate.tub.teadal:8000"
SELF_ADDRESSES	A , separated list of addresses this Advocate is reachable at.	["advocate.tub.teadal:8000","advocate.svc.cluster.local"]
ETH_POA	Type of consesus mechanism for settings with RPC	1 for POA, 0 for PoW/PoS
ETH_REQUEST_KWARGS_VERIFY	Verify secure communication with RPC	false doesn't verify, true verifies
INSECURE_HTTPS	Allows insecure requests with request library	false allows insecure, true doesn't allow insecure
ETH_GAS_PRICE	Fix gas price	if 0 then uses default
ETH_CHAINID	Chain ID of the blockchain network	1337
DATABASE_URL	The URL of the database to use for storing claims and other data.	"postgresql://admin:admin@localhost:5432/advocate"
FEDERATION_CONTRACT_ABI_PATH	The path to the federation contract ABI file.	"advocate/core/ethereum/contracts/BaseFederation.json"
FEDERATION_CONTRACT_ADDRESS	The address of the federation contract on the blockchain.	"0x0"
FEDERATION_NAME	The name of the federation contract.	"Test Federation"
FEDERATION_DESCRIPTION	The description of the federation contract.	"Test Federation for Advocate"
MOCKTNS_CONTRACT_ABI_PATH	The path to the mock TNS contract ABI file.	"advocate/core/ethereum/contracts/MockTNS.json"
ENS_CONTRACT_ADDRESS	The address of the ENS contract on the blockchain.	"0x0"
APPROVAL_THRESHOLD	Determines how many members of the federation must approve an action before it can be executed. (In percentage)	51
INTITAL_MEMBERS	A list of initial members of the federation. List of comma seperated eth addresses.	"0x1234567890abcdef1234567890abcdef12345678,0xabcdef1234567890abcdef1234567890abcdef12"
API_ADDRESS	The address of the REST API to use for the Advocate instance.	"127.0.0.1:8000"
TELEMETRY_INTERVAL	The interval in seconds to collect telemetry data from the Advocate instance.	60
PUBKEY_SERVICE_ADDRESS	The address public key fetching service used by advocate	"http://pubkey-service.advocate-testing.svc.cluster.local:5000/get_pubkey"
namespace_selector.zone	The namespaces Advocate monitors and interacts with.	["trust-plane", "default"]
evidence_sources.sources	The sources of telemetry and observabillity that Advocate is using.	["JaegerGRPC", "Kubernetes"]
logging.root_level	The root logging level for the Advocate instance.	"INFO"
logging.advocate_level	The logging level for advocate controller files and modules. (e.g. advocate.py, common.py, ...)	"DEBUG"
logging.telemetry_level	The logging level for telemetry collection and processing.	"DEBUG"
logging.claims_level	The logging level for claims processing, generation and storage.	"DEBUG"
logging.aggregator_level	The logging level for aggregators setup, processing and generation of aggregationClaims.	"DEBUG"
logging.rego_level	The logging level for Rego based aggregation.	"DEBUG"
logging.api_level	The logging level for the APIs of Advocate.	"DEBUG"
logging.ambassador_level	The logging level for the Ambassador (mutating webhook) of Advocate.	"DEBUG"

Besides the config, Advocate also requires a set of already deployed smart contracts. See the Smart Contracts documentation README in the Advocate Contracts repository for more information. Moreover, if you want to operate advocate inside Kubernetes you'll need to provide the deployed container a Kubernetes config or a service account with the required permissions to deploy the mutating webhook as well as the ability to read pods, jobs, deployments, and services.

Operation

Startup process of advocate

Advocate is deeply integrated within Kubernetes. When it is first started, it will need the authority to deploy a smart contract in the name of the Kubernetes operator. We recommend using a secret to store the private key of the operator. During startup, Advocate will first verify that the operator is a valid member of a TEADAL Federation before generating a node claim. This node claim functions as the identity document of this Kubernetes cluster afterward. We refer to it as the lake_id; with this identity, an operator can provide additional metadata information about the cluster, such as location, contact information, etc. This information is then stored in ipfs and anchored on the blockchain and can be used to verify the identity of the cluster. Once the identity is established, Advocate will install a mutating webhook (ambassador) into the Kubernetes cluster. This webhook will intercept all requests to the Kubernetes API for creating pods or jobs and inject the lake_id for each deployment as environment variables. This allows the pods to identify themselves to the advocate instance.

Runtime interactions of Advocate and newly started pods

During the runtime of Advocate, it intercepts the Kubernetes API for new pods and jobs. When a new pod is started, Advocate will inject the lake_id into the environment variables of the pod. Moreover, it will generate a new claim for this service and store it in IPFS. This claim contains the lake_id of the cluster, as well as information about the created deployment. The resulting hash of the claim is also attached as an environment variable to the pod. We refer to this as the entity_id. We require all services that allow for API interactions to log these environment variables as part of their output, e.g. when storing traces in a Jaeger. This way, we can later associate the traces with the entity that created them.

During the operation of any service tracked in this way, Advocate will constantly query available observability services, e.g., jaeger, to obtain traces of the service. We compile these traces into interaction claims that we also store in IPFS and, if required, in the document store. These claims can be used as evidence for the interactions of the service.

A Demonstration of Advocate's capabilities and deployment of additional tools is described in the Demo README.

An Example of processing FDPs and the respective tracking procedures performed by Advocate is present in the Data Pipeline README.

A Demonstration of Privacy tracking scenarios involving the Federation contract is described in the Privacy Scenarios README.

Variables injected into pods

Variable name	Description	Example
TEADAL_LAKE_ID	The lake_id of the cluster	"Buk5B"
TEADAL_ENTITY_ID	The entity_id of the the service	"F"
TEADAL_MANDATE	The mandate (claim) used to create the service (if available).	"0x0"

Each claim can be resolved to the corresponding IPFS hash or document store hash by decoding them using base58. Otherwise, the advocate instance provides a REST API to resolve them.

Further Documentation

What is the Aggreagator? How does the Aggregator work? How can I customize the Aggregation process? ...

• Aggregation and Rego: README

How can I setup and test all functionallities of Advocate in an Demonstrational environment? ...

• Advocate Demo: README

Which steps are needet to use the data pipeline and generate test data flow? ...

• Data Pipelines: README

How can I setup and test various privacy tracking scenarios using Advocate's Federation Contract? ...

• Privacy Scenarios: README

How can I deploy Advocate in a Kubernetes cluster? ...

• Deployment of Advocate: README

How do I setup and test the IPFS claims service? ...

• IPFS Testing: README

Data Model

Authors

Information Systems Engineering department of the TU Berlin for the TEADAL project.

License

This project is licensed under the Apache License, Version 2.0. For details, see the LICENSE.