Documentation updates

Author: anders-wartoft

README.md

@@ -240,6 +240,7 @@ export AIRGAP_UPSTREAM_TARGET_IP=255.255.255.255
 | sendingThreads           | AIRGAP_UPSTREAM_SENDING_THREADS | {"now": 0} | How many times, and when, whould we send each event? |
 | certFile                 | AIRGAP_UPSTREAM_CERT_FILE |               | For TLS to Kafka, add a certificate pem encoded file here |
 | keyFile                  | AIRGAP_UPSTREAM_KEY_FILE  |               | The private key for the certFile certificate |
+| keyPasswordFile     | AIRGAP_UPSTREAM_KEY_PASSWORD_FILE |       | Path to a file containing the password to decrypt an encrypted keyFile |
 | caFile                   | AIRGAP_UPSTREAM_CA_FILE   |               | The CA that issued the Kafka server's certificate |
 | deliverFilter            | AIRGAP_UPSTREAM_DELIVER_FILTER |               | Filter so not all events from Kafka is sent. Can be used to enable load balancing (see Load Balancing chapter below) |
 | topicTranslations        | AIRGAP_UPSTREAM_TOPIC_TRANSLATIONS |              | If you need to rename a topic before sending the messages you can use this: `{"inputTopic1":"outputTopic1","inputTopic2":"outputTopic2"}`. Here, if the name of a topic upstreams is `inputTopic1` it will be sent as `outputTopic1` from upstream |
@@ -282,6 +283,7 @@ The property privateKeyFiles should point to one or more private key files that
 | logFileName              | AIRGAP_DOWNSTREAM_LOG_FILE_NAME |          | If configured, all logs will be written to this file instead of the console. This will take effekt after the configuration is read and no errors occurs |
 | certFile                 | AIRGAP_DOWNSTREAM_CERT_FILE |               | For TLS to Kafka, add a certificate pem encoded file here |
 | keyFile                  | AIRGAP_DOWNSTREAM_KEY_FILE  |               | The private key for the certFile certificate |
+| keyPasswordFile     | AIRGAP_UPSTREAM_KEY_PASSWORD_FILE |       | Path to a file containing the password to decrypt an encrypted keyFile |
 | caFile                   | AIRGAP_DOWNSTREAM_CA_FILE   |               | The CA that issued the Kafka server's certificate |
 | internalTopic            | AIRGAP_DOWNSTREAM_INTERNAL_TOPIC    |  airgap-internal  | Topic name in Kafka to write to (internal logging). Topic name for events from the upstream topics will have the same name as the upstream topic, if not translated by the setting AIRGAP_DOWNSTREAM_TOPIC_TRANSLATIONS |
 | topicTranslations        | AIRGAP_DOWNSTREAM_TOPIC_TRANSLATIONS |            | Rename topics with a specified name to another name. Used in multi downstreams setup (see Redundancy and Load Balancing.md) |

doc/Deduplication.md

@@ -3,35 +3,43 @@
 ## Installation
 
 1. **Clone the repository:**
-	```bash
-	git clone https://github.com/anders-wartoft/air-gap.git
-	cd air-gap
-	```
-
-2. **Build the binaries:**
-	```bash
-	make all
-	```
-	This builds upstream, downstream, and the deduplication Java application.
-
-3. **Install dependencies:**
-	- Go 1.18+ (for upstream/downstream)
-	- Java 17+ (for deduplication)
-	- Kafka 3.9+ (for event streaming)
-	- Optional: Metricbeat, Jolokia for monitoring
-
-4. **Prepare configuration files:**
-	- Copy and edit example configs in `config/` and `config/testcases/`.
-	- See below for details.
-
-5. **Generate keys for encryption (optional):**
-	- See README.md section "Keys" for key generation commands.
+
+```bash
+git clone https://github.com/anders-wartoft/air-gap.git
+cd air-gap
+```
+
+2\. **Build the binaries:**
+
+```bash
+make all
+```
+
+This builds upstream, downstream, and the deduplication Java application.
+
+3\. **Install dependencies:**
+
+- Go 1.18+ (for upstream/downstream)
+- Java 17+ (for deduplication)
+- Kafka 3.9+ (for event streaming)
+- Optional: Metricbeat, Jolokia for monitoring
+
+4\. **Prepare configuration files:**
+
+- Copy and edit example configs in `config/` and `config/testcases/`.
+- See below for details.
+
+5\. **Generate keys for encryption (optional):**
+
+- See README.md section "Keys" for key generation commands.
 
 ## Configuration
 
 ### Upstream
+
 Edit your upstream config file (e.g., `config/upstream.properties`):
-```properties
+
+```bash
 id=Upstream_1
 nic=en0
 targetIP=127.0.0.1
@@ -44,11 +52,14 @@ publicKeyFile=certs/server2.pem
 generateNewSymmetricKeyEvery=500
 mtu=auto
 ```
+
 Override any setting with environment variables (see README for details).
 
 ### Downstream
+
 Edit your downstream config file (e.g., `config/downstream.properties`):
-```properties
+
+```bash
 id=Downstream_1
 nic=en0
 targetIP=0.0.0.0
@@ -62,8 +73,10 @@ clientId=downstream
 ```
 
 ### Deduplicator (Java)
+
 Edit your deduplication config (e.g., `config/create.properties`):
-```properties
+
+```bash
 RAW_TOPICS=transfer
 CLEAN_TOPIC=dedup
 GAP_TOPIC=gaps
@@ -79,42 +92,55 @@ APPLICATION_ID=dedup-gap-app
 For details on how to run the applications as services, see README.md
 
 #### Performance Tuning
+
 For high event rates (e.g., 10,000 eps):
+
 - `PERSIST_INTERVAL_MS`: 100–1000 ms (persist state every 0.1–1 second)
 - `COMMIT_INTERVAL_MS`: 100–1000 ms (commit progress every 0.1–1 second)
+
 Start with:
-```
+
+```bash
 PERSIST_INTERVAL_MS=500
 COMMIT_INTERVAL_MS=500
 ```
+
 This means state and progress are checkpointed every 0.5 seconds, so at most 5,000 events would need to be re-processed after a crash.
 
 **Tuning tips:**
+
 - Lower values = less data loss on crash, but more I/O.
 - Higher values = less I/O, but more data to reprocess after a failure.
 - Monitor RocksDB and Kafka broker load; adjust if you see bottlenecks.
 
 ## Running the Applications
 
 **Upstream:**
+
 ```bash
 go run src/cmd/upstream/main.go config/upstream.properties
 ```
+
 or (after build):
+
 ```bash
 ./src/cmd/upstream/upstream config/upstream.properties
 ```
 
 **Downstream:**
+
 ```bash
 go run src/cmd/downstream/main.go config/downstream.properties
 ```
+
 or (after build):
+
 ```bash
 ./src/cmd/downstream/downstream config/downstream.properties
 ```
 
 **Deduplicator:**
+
 ```bash
 java -jar java-streams/target/air-gap-deduplication-fat-<version>.jar
 ```
@@ -135,58 +161,66 @@ See `doc/Monitoring.md` for instructions on using Metricbeat, Jolokia, and JMX f
 To completely uninstall air-gap and its components:
 
 1. **Stop running services:**
-	```bash
-	sudo systemctl stop upstream.service
-	sudo systemctl stop downstream.service
-	sudo systemctl stop dedup.service
-	```
-
-2. **Disable services:**
-	```bash
-	sudo systemctl disable upstream.service
-	sudo systemctl disable downstream.service
-	sudo systemctl disable dedup.service
-	```
-
-3. **Remove binaries:**
-	```bash
-	rm -f /opt/airgap/upstream/bin/*
-	rm -f /opt/airgap/downstream/bin/*
-	rm -f /opt/airgap/dedup/bin/*
-	rm -f /usr/local/bin/upstream
-	rm -f /usr/local/bin/downstream
-	rm -f /usr/local/bin/dedup
-	```
-
-4. **Remove configuration files:**
-	```bash
-	rm -rf /opt/airgap/upstream/*.properties
-	rm -rf /opt/airgap/downstream/*.properties
-	rm -rf /opt/airgap/dedup/*.properties
-	rm -rf /etc/airgap/
-	```
-
-5. **Remove keys and certificates (if used):**
-	```bash
-	rm -rf /opt/airgap/certs/
-	```
-
-6. **Remove systemd service files:**
-	```bash
-	sudo rm -f /etc/systemd/system/upstream.service
-	sudo rm -f /etc/systemd/system/downstream.service
-	sudo rm -f /etc/systemd/system/dedup.service
-	sudo systemctl daemon-reload
-	```
-
-7. **Remove log files:**
-	```bash
-	rm -rf /var/log/airgap/
-	```
-
-8. **(Optional) Remove cloned source directory:**
-	```bash
-	rm -rf ~/air-gap
-	```
+
+```bash
+sudo systemctl stop upstream.service
+sudo systemctl stop downstream.service
+sudo systemctl stop dedup.service
+```
+
+2\. **Disable services:**
+
+```bash
+sudo systemctl disable upstream.service
+sudo systemctl disable downstream.service
+sudo systemctl disable dedup.service
+```
+
+3\. **Remove binaries:**
+
+```bash
+rm -f /opt/airgap/upstream/bin/*
+rm -f /opt/airgap/downstream/bin/*
+rm -f /opt/airgap/dedup/bin/*
+rm -f /usr/local/bin/upstream
+rm -f /usr/local/bin/downstream
+rm -f /usr/local/bin/dedup
+```
+
+4\. **Remove configuration files:**
+
+```bash
+rm -rf /opt/airgap/upstream/*.properties
+rm -rf /opt/airgap/downstream/*.properties
+rm -rf /opt/airgap/dedup/*.properties
+rm -rf /etc/airgap/
+```
+
+5\. **Remove keys and certificates (if used):**
+
+```bash
+rm -rf /opt/airgap/certs/
+```
+
+6\. **Remove systemd service files:**
+
+```bash
+sudo rm -f /etc/systemd/system/upstream.service
+sudo rm -f /etc/systemd/system/downstream.service
+sudo rm -f /etc/systemd/system/dedup.service
+sudo systemctl daemon-reload
+```
+
+7\. **Remove log files:**
+
+```bash
+rm -rf /var/log/airgap/
+```
+
+8\. **(Optional) Remove cloned source directory:**
+
+```bash
+rm -rf ~/air-gap
+```
 
 **Note:** Adjust paths as needed for your installation. If you installed to custom locations, remove those as well.