Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions services/scheduled_parse_prove_feed/.env-example
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ETH_WALLET_PRIVATE_KEY=PUT_YOUR_ETH_WALLET_PRIVATE_KEY_HERE
ALCHEMY_API_KEY=PUT_YOUR_ETH_WALLET_PRIVATE_KEY_HERE
55 changes: 55 additions & 0 deletions services/scheduled_parse_prove_feed/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Diffuse Datafeed (systemd service + timer)

This project allows you to schedule the `parse_and_prove_feed.sh` script using `systemd`. The script, in turn, runs the Python program `parse_prove_feed.py` with parameters from the `.env` file and writes logs to `/var/log/diffuse/parse_and_prove_feed.log`.

## Installation

1. Place the project files in the desired directory, for example: `~/datafeed-cli/`.
2. Create a `.env` file based on `.env-example` and fill in the required variables.
3. In the files `diffuse-parse-proove-feed.service` and `diffuse-parse-proove-feed.timer`, specify:
- The absolute paths to the directory containing the scripts and `.env`.
- If needed, adjust the timer frequency (by default, every 5 seconds).
4. In `parse_and_prove_feed.sh`, specify any parameters needed for `parse_prove_feed.py`.
5. Run the setup script:

```bash
chmod +x run.sh
./run.sh
```

This script copies the relevant files to the system directories, reloads `systemd`, and enables the timer.

## Checking Status

- To check the timer status:

```bash
sudo systemctl status diffuse-parse-proove-feed.timer
```

- To view logs:

```bash
tail -f /var/log/diffuse/parse_and_prove_feed.log
```

## Stopping

To stop the service and timer:

```bash
./stop.sh
```

This script will stop and disable both the service and the timer (check with `systemctl status diffuse-parse-proove-feed.timer`).

## Changing the Schedule

To change the run frequency, edit `OnUnitActiveSec` and other settings in `diffuse-parse-proove-feed.timer`, then run:

```bash
sudo systemctl daemon-reload
sudo systemctl restart diffuse-parse-proove-feed.timer
```

All logs will continue to be written to `/var/log/diffuse/parse_and_prove_feed.log`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[Unit]
Description=Diffuse Datafeed periodic job

[Service]
Type=oneshot
User=PUT_YOUR_USER_HERE
WorkingDirectory=PUT_YOUR_ABSOLUT_PATH_TO_DATAFEED_CLI_HERE ;
EnvironmentFile=PUT_YOUR_ABSOLUT_PATH_TO_DATAFEED_CLI_HERE/services/scheduled_parse_prove_feed/.env ;
ExecStart=PUT_YOUR_ABSOLUT_PATH_TO_DATAFEED_CLI_HERE/services/scheduled_parse_prove_feed/parse_and_prove_feed.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[Unit]
Description=Runs Diffuse Datafeed job every 5 seconds, sequentially
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too much, lets change to 86400


[Timer]
OnBootSec=5
OnUnitActiveSec=5
AccuracySec=1s
Persistent=false

[Install]
WantedBy=timers.target
9 changes: 9 additions & 0 deletions services/scheduled_parse_prove_feed/parse_and_prove_feed.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash

mkdir -p /var/log/diffuse

echo "$(date) start feeding" >> /var/log/diffuse/parse_and_prove_feed.log

python3 cli/parse_prove_feed.py PUT_YOUR_CLI_PARAMETERS_HERE >> /var/log/diffuse/parse_and_prove_feed.log 2>&1

echo "$(date) data request and proving finished: " >> /var/log/diffuse/parse_and_prove_feed.log
17 changes: 17 additions & 0 deletions services/scheduled_parse_prove_feed/run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env bash

sudo mkdir -p /var/log/diffuse

sudo cp diffuse-parse-proove-feed.service /etc/systemd/system/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this design works well with exactly one task, but what if we want to have for example two in parallel with two different heartbeats?
every 10 minutes with --pairs_file_path=subset_600_sec.txt
every day with --pairs_file_path=subset_86400_sec.txt

sudo cp diffuse-parse-proove-feed.timer /etc/systemd/system/

sudo chmod +x PUT_YOUR_ABSOLUT_PATH_TO_DATAFEED_CLI_HERE/services/scheduled_parse_prove_feed/parse_and_prove_feed.sh

sudo systemctl daemon-reload

sudo systemctl enable diffuse-parse-proove-feed.timer
sudo systemctl start diffuse-parse-proove-feed.timer

echo "System-wide service and timer have been installed and started (running under root)."
echo "Check status via: sudo systemctl status diffuse-parse-proove-feed.timer"
echo "Logs will be written to: /var/log/diffuse/parse_and_prove_feed.log"
10 changes: 10 additions & 0 deletions services/scheduled_parse_prove_feed/stop.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash

systemctl stop diffuse-parse-proove-feed.timer
systemctl disable diffuse-parse-proove-feed.timer

systemctl stop diffuse-parse-proove-feed.service
systemctl disable diffuse-parse-proove-feed.service

echo "Timer and service 'diffuse-parse-proove-feed' have been stopped and disabled."
echo "You can verify via: systemctl status diffuse-parse-proove-feed.timer"