Dev Setup

Everything is in docker now. So, install docker. Then build the dev container as follows:

docker build -f docker/build.dockerfile . -t psl_builder

Download the current set of third-party libraries from https://thirdpartydependencies.blob.core.windows.net/third-party/third-party.zip and extract into third-party directory.

Build using:

mkdir -p build
docker run -t -v .:/psl_kv psl_builder

Dev Runtime

Dev runtime is based on docker compose (or docker-compose, whichever you prefer)

In the docker directory, run:

docker compose build
docker compose up
# You can use separate terminals for each of {worker, sequencer, dc, client, forwarder}.

Prod runtime

Production runtime is managed by Kubernetes. We use k3s to create a cluster on top of separately allocated VMs. Azure-managed kubernetes doesn't have Linux kernel 6+ needed for SGX2, neither does it allow us to create a geo-distributed setup.

Infra setup

Install Azure cli and Terraform.

Run az login.

Clone https://github.com/data-capsule/terraform-configs-azure

Now in that repo, run the following:

terraform plan -out main.tfplan
terraform apply

This allocates all the required VMs.

Now setup kubernetes by running:

sh k3s-install.sh

Cluster manager will be in the VM named storagepool_vm0. Find out the ip address of the VM from Azure.

Get the kubectl config from storagepool_vm0 as follows:

ssh -i privkey.pem azureadmin@ip_of_storagepool_vm0 'sudo cat /etc/rancher/k3s/k3s.yaml'

To be able to kubectl into the cluster from your dev machine, run the following:

snap install kubectl --classic  # Assuming using Ubuntu with snap installed
mkdir -p ~/.kube
export KUBECTL='kubectl --insecure-skip-tls-verify' # Put this line is bashrc

Copy the kubectl config over to ~/.kube/config in your dev machine. Change the ip address in the server field with the ip of storagepool_vm0.

Verify the setup is correct:

$KUBECTL get nodes

Lastly, we need to label the nodes, so that the node affinities in the kubernetes configs are satisfied. For that, run:

$KUBECTL get nodes | grep storage | cut -d' ' -f 1 | xargs -I{} sudo $KUBECTL label nodes {} nodegroup=storage
$KUBECTL get nodes | grep sgx | cut -d' ' -f 1 | xargs -I{} sudo $KUBECTL label nodes {} nodegroup=sgx
$KUBECTL get nodes | grep sev | cut -d' ' -f 1 | xargs -I{} sudo $KUBECTL label nodes {} nodegroup=sev

We now need to connect to our container registry in both docker and kubernetes. We use a container registry called psldeployment.azurecr.io. This is hard-coded for now. For this, run:

docker login psldeployment.azurecr.io
# Username: psldeployment
# Password: Get from Home > Container registries > psldeployment in Azure website

cat ~/.docker/config.json   # Make sure the file exists

$KUBECTL create secret generic regcred \
    --from-file=.dockerconfigjson=~/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

Start the prometheus and grafana servers for monitoring as follows:

cd manager/k8s-configs/prometheus
$KUBECTL create namespace monitoring
$KUBECTL apply -f .
$KUBECTL get pods -w -n monitoring      # Wait till all of the pods are Running.

Infra teardown

From terraform-azure-configs repo, run:

terraform destroy

Please destroy the infra daily or more frequently, otherwise the costs are going to be very high. Need to rerun the whole infra setup phase for rebuilding the infra.

Running

In the manager/k8s-configs run make.

To stop, run make clean.

For safely applying updates to code, build using the psl_builder dev container and then in the manager/k8s-configs run: make clean && make Occassionally, run docker buildx prune to clean up remnants of old images.

The client-1 pod runs a web server waiting for you to upload a wasm binary and a benchmark config. To access it, on a separate terminal run:

$KUBECTL port-forward pods/client-1 4001:4001 --address '0.0.0.0'

To upload a wasm binary, run:

curl -X 'POST' 'http://localhost:4001/upload/' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-F 'file=@path/to/wasm;type=application/wasm'

Visit http://ip_of_dev_machine:4001 to get a frontend to upload benchmark config and view running logs.

To visit the Grafana dashboard, on another separate terminal run:

$KUBECTL port-forward -n monitoring svc/grafana-service 3000:3000 --address='0.0.0.0'

and vist http://ip_of_dev_machine:3000. (Default credentials are admin:admin)