From 3bd9840c6cdd98f849610f0751ba0e1a853a1678 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 28 Jul 2023 00:00:05 +0200 Subject: [PATCH 1/6] [Development] Add LocalDev pod debugging --- build/localnet/README.md | 26 ++++++++++++++++++++++++++ build/localnet/Tiltfile | 15 ++++++++++++--- 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/build/localnet/README.md b/build/localnet/README.md index ae52c319a..425ebc4eb 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -206,6 +206,32 @@ The k8s manifests that `tilt` submits to the cluster can be found in [this direc Tilt continuously monitors files on local filesystem in [specific directories](Tiltfile#L27), and it rebuilds the binary and distributes it to the pods on every code change. This allows developers to iterate on the code and see the changes immediately (i.e. hot-reloading). +## Connect to a pod debugging server + +LocalNet pocket nodes have a dlv debugging server opened on port 7081. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel with the pod we want to debug. + +``` +kubectl port-forward validator-001-pocket-0 7081:7081 +``` + +### Configure VSCode debugger + +Add the following to VSCode `launch.json` configuration + +``` +{ + "name": "Debug Pocket Node", + "type": "go", + "request": "attach", + "mode": "remote", + "remotePath": "${workspaceFolder}", + "port": 7081, + "host": "localhost" +} +``` + +Then you have just to set your breakpoints and start the debugging session (no need to rebuild or restart the binary). + ## Troubleshooting ### Why? diff --git a/build/localnet/Tiltfile b/build/localnet/Tiltfile index 9f5a36b4c..0b284a110 100644 --- a/build/localnet/Tiltfile +++ b/build/localnet/Tiltfile @@ -102,15 +102,24 @@ local_resource( docker_build_with_restart( "pocket-image", root_dir, - dockerfile_contents="""FROM debian:bullseye + dockerfile_contents="""FROM golang:1.20-bullseye +RUN go install github.com/go-delve/delve/cmd/dlv@latest COPY bin/pocket-linux /usr/local/bin/pocket WORKDIR / """, only=["./bin/pocket-linux"], entrypoint=[ + "dlv", + "exec", "/usr/local/bin/pocket", - "-config=/pocket/configs/config.json", - "-genesis=/pocket/configs/genesis.json", + "--headless", + "--listen=:7081", + "--api-version=2", + "--accept-multiclient", + "--continue", + "--", + "--config=/pocket/configs/config.json", + "--genesis=/pocket/configs/genesis.json", ], live_update=[sync("bin/pocket-linux", "/usr/local/bin/pocket")], ) From 0960d4855c03df873b86c83f0c25617749345867 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 28 Jul 2023 01:40:36 +0200 Subject: [PATCH 2/6] [Documentation] Add breakpoint video --- build/localnet/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/build/localnet/README.md b/build/localnet/README.md index 425ebc4eb..03bc799c8 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -232,6 +232,9 @@ Add the following to VSCode `launch.json` configuration Then you have just to set your breakpoints and start the debugging session (no need to rebuild or restart the binary). +[Watch demo](https://github.com/pokt-network/pocket/assets/231488/3525161f-2098-488a-8f36-8747e40320a6) + + ## Troubleshooting ### Why? From 3887c5d3660527460488577f33184ec36fc3d814 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 28 Jul 2023 12:46:55 +0200 Subject: [PATCH 3/6] Update README.md --- build/localnet/README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/build/localnet/README.md b/build/localnet/README.md index 03bc799c8..eb588b8b7 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -206,15 +206,17 @@ The k8s manifests that `tilt` submits to the cluster can be found in [this direc Tilt continuously monitors files on local filesystem in [specific directories](Tiltfile#L27), and it rebuilds the binary and distributes it to the pods on every code change. This allows developers to iterate on the code and see the changes immediately (i.e. hot-reloading). -## Connect to a pod debugging server +## Debug with dlv -LocalNet pocket nodes have a dlv debugging server opened on port 7081. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel with the pod we want to debug. +### k8s LocalNet: Connect to a node debugging server + +LocalNet Pocket nodes have a dlv debugging server opened on port 7081. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel with the pod we want to debug. ``` kubectl port-forward validator-001-pocket-0 7081:7081 ``` -### Configure VSCode debugger +#### Configure VSCode debugger Add the following to VSCode `launch.json` configuration @@ -234,6 +236,9 @@ Then you have just to set your breakpoints and start the debugging session (no n [Watch demo](https://github.com/pokt-network/pocket/assets/231488/3525161f-2098-488a-8f36-8747e40320a6) +### docker-compose based LocalNet + +### Debug tests ## Troubleshooting From 4dee0079d187e03edc8f77c311f4edb35b207696 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 4 Aug 2023 09:43:59 +0200 Subject: [PATCH 4/6] add make kube_debug command --- Makefile | 7 +++++++ build/localnet/README.md | 17 ++++++++++++----- 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 9ec1c636a..f77b7f3d7 100644 --- a/Makefile +++ b/Makefile @@ -625,3 +625,10 @@ ggshield_secrets_scan: ## Scans the project for secrets using ggshield .PHONY: ggshield_secrets_add ggshield_secrets_add: ## A helper that adds the last results from `make ggshield_secrets_scan`, store in `.cache_ggshield` to `.gitguardian.yaml`. See `ggshield for more configuratiosn` ggshield secret ignore --last-found + +# The validator's pod name to debug +debugPod=validator-001-pocket-0 + +.PHONY: kube_debug +kube_debug: ## Forward dlv client connections to debugPod that should be running a dlv server at the same port + kubectl port-forward ${debugPod} 7081:7081 diff --git a/build/localnet/README.md b/build/localnet/README.md index eb588b8b7..6faafb153 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -16,9 +16,16 @@ This guide shows how to deploy a LocalNet using [pocket-operator](https://github - [Interacting w/ LocalNet](#interacting-w-localnet) - [Make Targets](#make-targets) - [Addresses and keys on LocalNet](#addresses-and-keys-on-localnet) + - [Applications staked on LocalNet](#applications-staked-on-localnet) + - [Servicers staked on LocalNet](#servicers-staked-on-localnet) - [How to change configuration files](#how-to-change-configuration-files) - [Overriding default values for localnet with Tilt](#overriding-default-values-for-localnet-with-tilt) - [How does it work?](#how-does-it-work) +- [Debug with dlv](#debug-with-dlv) + - [k8s LocalNet: Connect to a node debugging server](#k8s-localnet-connect-to-a-node-debugging-server) + - [Configure VSCode debugger](#configure-vscode-debugger) + - [docker-compose based LocalNet](#docker-compose-based-localnet) + - [Debug tests](#debug-tests) - [Troubleshooting](#troubleshooting) - [Why?](#why) - [Force Trigger an Update](#force-trigger-an-update) @@ -210,17 +217,17 @@ Tilt continuously monitors files on local filesystem in [specific directories](T ### k8s LocalNet: Connect to a node debugging server -LocalNet Pocket nodes have a dlv debugging server opened on port 7081. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel with the pod we want to debug. +LocalNet Pocket nodes have a `dlv` debugging server opened on port `7081`. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel to the pod we want to debug. -``` -kubectl port-forward validator-001-pocket-0 7081:7081 +```bash +make kube_config ``` #### Configure VSCode debugger -Add the following to VSCode `launch.json` configuration +Add the following to your VSCode debugging configuration file `.vscode/launch.json`. You can add the file from IDE if it does not exist. -``` +```json { "name": "Debug Pocket Node", "type": "go", From 14169c2ac08604281306ba8265222341d9ab58ac Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 4 Aug 2023 09:48:39 +0200 Subject: [PATCH 5/6] add VSCode screenshot --- build/localnet/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/build/localnet/README.md b/build/localnet/README.md index 6faafb153..b4d5d207f 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -227,6 +227,8 @@ make kube_config Add the following to your VSCode debugging configuration file `.vscode/launch.json`. You can add the file from IDE if it does not exist. +![image](https://github.com/pokt-network/pocket/assets/231488/2882008e-3632-469c-9d1e-2f77ad785dfe) + ```json { "name": "Debug Pocket Node", From 8d6318ae883789cee2590f63b4b7b138d9135450 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 4 Aug 2023 10:19:24 +0200 Subject: [PATCH 6/6] add a workaround for halted pod restart --- build/localnet/README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/build/localnet/README.md b/build/localnet/README.md index b4d5d207f..6635fba77 100644 --- a/build/localnet/README.md +++ b/build/localnet/README.md @@ -24,7 +24,6 @@ This guide shows how to deploy a LocalNet using [pocket-operator](https://github - [Debug with dlv](#debug-with-dlv) - [k8s LocalNet: Connect to a node debugging server](#k8s-localnet-connect-to-a-node-debugging-server) - [Configure VSCode debugger](#configure-vscode-debugger) - - [docker-compose based LocalNet](#docker-compose-based-localnet) - [Debug tests](#debug-tests) - [Troubleshooting](#troubleshooting) - [Why?](#why) @@ -245,7 +244,20 @@ Then you have just to set your breakpoints and start the debugging session (no n [Watch demo](https://github.com/pokt-network/pocket/assets/231488/3525161f-2098-488a-8f36-8747e40320a6) -### docker-compose based LocalNet +k8s runs liveness and readiness checks will restart the pods that are not responsive. In order to not have the debugging session dropped because of the pod restarting, you have to comment `livenessProbe` and `readinessProbe` sections in `charts/pocket/templates/statefulset.yaml` + +```yaml +# Comment this section disable liveness checks for debugging +livenessProbe: + httpGet: + path: /v1/health + port: rpc +# Comment this section disable readiness checks for debugging +readinessProbe: + httpGet: + path: /v1/health + port: rpc +``` ### Debug tests