Jjschwartz · Namron2000 · Jan 16, 2025
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
diff --git a/README.rst b/README.rst
@@ -79,6 +79,16 @@ License
 What's new
 ----------
 
+- 2024-11-01 (v 0.12.1) (MAJOR release, WOP)
+
+  - Modified version of NASim based on the paper "Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN" `[Norman Becker et al.] <https://arxiv.org/abs/2407.15656>`_. Please refer to the documentation for further information
+
+  + Addition of Credentials and Vulnerabilities
+  + Modifications to existing functions and structures to reflect the intended
+  + Modifications to the generator in regards to the new additions
+
+
+
 
 - 2023-05-14 (v 0.12.0) (MINOR release)
 

diff --git a/docs/Makefile b/docs/Makefile
diff --git a/docs/make.bat b/docs/make.bat
@@ -1,35 +1,35 @@
-@ECHO OFF
-
-pushd %~dp0
-
-REM Command file for Sphinx documentation
-
-if "%SPHINXBUILD%" == "" (
-	set SPHINXBUILD=sphinx-build
-)
-set SOURCEDIR=source
-set BUILDDIR=build
-
-if "%1" == "" goto help
-
-%SPHINXBUILD% >NUL 2>NUL
-if errorlevel 9009 (
-	echo.
-	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
-	echo.installed, then set the SPHINXBUILD environment variable to point
-	echo.to the full path of the 'sphinx-build' executable. Alternatively you
-	echo.may add the Sphinx directory to PATH.
-	echo.
-	echo.If you don't have Sphinx installed, grab it from
-	echo.http://sphinx-doc.org/
-	exit /b 1
-)
-
-%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
-goto end
-
-:help
-%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
-
-:end
-popd
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/docs/requirements.txt b/docs/requirements.txt
diff --git a/docs/source/community/acknowledgements.rst b/docs/source/community/acknowledgements.rst
diff --git a/docs/source/community/development.rst b/docs/source/community/development.rst
diff --git a/docs/source/community/distributing.rst b/docs/source/community/distributing.rst
diff --git a/docs/source/community/index.rst b/docs/source/community/index.rst
diff --git a/docs/source/conf.py b/docs/source/conf.py
diff --git a/docs/source/explanations/index.rst b/docs/source/explanations/index.rst
diff --git a/docs/source/explanations/scenario_generation.rst b/docs/source/explanations/scenario_generation.rst
@@ -57,3 +57,47 @@ Firstly, there exists no firewall between subnets in the user zone. So communica
 Secondly, the number of services blocked is controlled by the ``restrictiveness`` parameter. This controls the number of services to block between zones (i.e. between the internet, DMZ, sensitive, and user zones).
 
 Thirdly, to ensure that the goal can be reached, traffic from at least one service running on each subnet will be allowed between each zone. This may mean more services will be allowed than restrictiveness parameter.
+
+
+Credentials (Optional)
+----------------------
+
+To include credentials the following arguments are introduced:
+
+- ``num_cred=0`` - represents the number of credentials used. (max. of 9)
+
+- ``wiretapping_cost=0`` - cost to wiretap a target
+
+In general every exploit can be used with each credential, each exploit is generated with every possible credential as an input. These permutations are handled as one exploit to not affect the ``num_exploits=None`` variable.
+
+**Distribution**: Each credentials only has singular source to be discovered on. They are randomly handed to each subnet and then further distributed to the hosts in each subnet. 
+
+Since every Host is vulnerable to at least one exploit, the credentials are placed in the ``credentials_tofind`` variable of the host and can be found by ``Wiretapping()``.
+
+**Locking**: Following the distribution the hosts are potentionaly locked. For this a random order to visit each host is generated and an array of ``found_credentials`` is being tracked.
+
+Each visited host is locked with a random already 'found' credential (if possible), credentials saved in ``credentials_tofind`` are noted for future use.
+
+**Limitations**: 'Distribution' and 'Locking' are simple representations of the potential applications.
+
+- Both are fully random and not influenced by ``seed`` or ``restrictiveness``
+
+- credentials found by ``PrivelegeEscalation()`` are not represented nor implemented
+
+- 'Locking' does not uniformally 'lock' existing hosts, resulting in a random density of this property
+
+
+Vulnerabilities (Optional)
+--------------------------
+
+- ``num_vul=0`` - number of vulnerabilities
+
+- ``vul_scan_cost=0`` - cost to scan a host for vulnerabilities
+
+Similiar to the depiction of how vulnerabilities work, each one is linked to a service. The exploits utilizing these vulnerabilities are not bound to a OS and have a ``prob=1``.
+
+Hosts running a 'vulnerable' service also get the vulnerability alocated to them to ensure susceptibility.
+
+**Limitations**: Vulnerabilities are randomly bound to services and not additionally to OS's, making it less diverse in terms of permutations.
+
+Furthermore, the possibility of a service with a linked vulnerability running on a host without the vulnerability is not present.
diff --git a/docs/source/explanations/sim_to_real.rst b/docs/source/explanations/sim_to_real.rst
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -9,6 +9,16 @@ The environment is modelled after the `gymnasium (formerly Open AI gym) <https:/
 What's new
 ----------
 
+
+Version 0.12.1
+**************
+
+  + Modified version of NASim based on the paper "Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN" `[Norman Becker et al.] <https://arxiv.org/abs/2407.15656>`_. Please refer to the documentation for further information
+
+  - Addition of Credentials and Vulnerabilities
+  - Modifications to existing functions and structures to reflect the intended
+  - Modifications to the generator in regards to the new additions
+
 Version 0.12.0
 **************
 

diff --git a/docs/source/reference/agents/index.rst b/docs/source/reference/agents/index.rst
diff --git a/docs/source/reference/envs/actions.rst b/docs/source/reference/envs/actions.rst
diff --git a/docs/source/reference/envs/environment.rst b/docs/source/reference/envs/environment.rst
diff --git a/docs/source/reference/envs/host_vector.rst b/docs/source/reference/envs/host_vector.rst
diff --git a/docs/source/reference/envs/index.rst b/docs/source/reference/envs/index.rst
diff --git a/docs/source/reference/envs/observation.rst b/docs/source/reference/envs/observation.rst
diff --git a/docs/source/reference/envs/state.rst b/docs/source/reference/envs/state.rst
diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst
diff --git a/docs/source/reference/load.rst b/docs/source/reference/load.rst
diff --git a/docs/source/reference/scenarios/benchmark_scenarios.rst b/docs/source/reference/scenarios/benchmark_scenarios.rst
diff --git a/docs/source/reference/scenarios/benchmark_scenarios_agent_scores.csv b/docs/source/reference/scenarios/benchmark_scenarios_agent_scores.csv
diff --git a/docs/source/reference/scenarios/benchmark_scenarios_table.csv b/docs/source/reference/scenarios/benchmark_scenarios_table.csv
diff --git a/docs/source/reference/scenarios/generator.rst b/docs/source/reference/scenarios/generator.rst
diff --git a/docs/source/reference/scenarios/index.rst b/docs/source/reference/scenarios/index.rst
diff --git a/docs/source/tutorials/creating_scenarios.rst b/docs/source/tutorials/creating_scenarios.rst
diff --git a/docs/source/tutorials/credentials_vulnerabilities.rst b/docs/source/tutorials/credentials_vulnerabilities.rst
@@ -0,0 +1,153 @@
+.. _`cred_vul_tute`:
+
+Credentials and Vulnerabilities
+===============================
+
+The following additions are optional and do not need to be defined in costum scenario.yaml files. Nevertheless, internal structures had to be modified which are 'visible' even if not manually defined.
+
+
+Idea
+----
+
+To better reflect real world complexity, credentials and vulnerabilities are introduced to NASim.
+
+These modifications are implemented with the idea to train RL-agents in more realistic enviroments.
+
+
+Credentials
+-----------
+
+Credentials are implemented as single digits from **1** to **9**, each describing a full credential pair of, theoretically, a username and password. **0** is in meaning equivilant to ``None``.
+
+
+Host
+^^^^
+
+Hosts can now be further described by ``credentials_needed: int`` and ``credentials_tofind: int``.
+
+.. code-block:: yaml
+
+    host_configurations:
+        (1, 0):
+            os: linux
+            services: [ssh]
+            processes: [tomcat]
+            firewall:
+                (3, 0): [ssh]
+            value: 0
+            credentials_needed: 0
+            credentials_tofind: 12
+
+``credentials_needed: int`` describes the credentials needed to perform a succesful exploit. If 0 or not defined no credentials are needed.
+
+``credentials_tofind: int`` describes the credentials which can be found by wiretapping from a specific host.
+This is a concatination of integer, meaning `123` is equal to its permutations for example `213`.
+
+
+PrivelegeEscalation
+^^^^^^^^^^^^^^^^^^^
+
+Under the pretext of post-exploitation, **PrivelegeEscalation()** can be further extended with ``credentials_tofind: int``. 
+
+.. code-block:: yaml
+
+    privilege_escalation:
+        pe_gain_Credentails_from_Shadow:
+            process: shadow
+            os: Rasbian
+            prob: 1.0
+            cost: 3
+            access: root
+            credentials_tofind: 3
+
+``credentials_tofind: int`` describes the credentials which are found by exploiting / attacking a process on the specific host.
+
+
+Wiretapping
+^^^^^^^^^^^
+
+
+**Wiretapping()** is a new action with the goal to 'listen' from a targeted node. A compromised host can tap to reveal the credentials hold in ``cred_tofind: int``.
+
+The cost ``wiretapping_cost: int`` can be set in the scenario.yaml.
+
+
+
+Vulnerabilities
+---------------
+
+A vulnerability is a weakness or flaw on a host which can be exploited.
+
+It is implemented as a ``string`` and with its introduction both **Host** and **Exploit** get additional options and a new action **VulScan()** is implemented.
+
+
+Host
+^^^^
+
+.. code-block:: yaml
+
+    host_configurations:
+        (1, 0):
+            os: Rasbian
+            services: [openssh, samba] 
+            vul: [CVE20072447]
+            processes: [shadow]
+            credentials_needed: 0
+            credentials_tofind: 21
+
+``vul: list[str]`` describes the existing vulnerabilities on a host.
+
+
+Exploit
+^^^^^^^
+
+.. code-block:: yaml
+
+    e_samba:
+        service: samba
+        os: None
+        vul: CVE20072447
+        prob: 1.0
+        cost: 3
+        access: root
+
+``vul: str`` describes the exploited vulnerability. This `vul` has to exist on the targeted host for the exploit to succeed.
+Can further be *None* or not defined if not needed.
+
+
+VulScan
+^^^^^^^
+
+Similiar to **OSScan()** it is an action performed on a host returning its vulnerabilites ``vul: list[str]``.
+
+The cost ``vul_scan_cost: int`` can be set in the scenario.yaml.
+
+
+Scenarios
+^^^^^^^^^
+
+Some scenarios are added featuring credentials and vulnerabilities, namely *tiny-wire*, *tiny-post*, *tiny-cred*, *small-wire* and *small-post*.
+
+In addition, some scenarios from previous work got an equivalant *.yaml* with credentials and vulnerabilities specifically defined. (names ending on *-cwp*) 
+
+
+Information
+-----------
+
+If no credentials or vulnerabilities are used, following properties are "visible":
+
+- **HostVector** will still hold ``cred_tofind = 0`` and ``cred_found = 0``.
+
+- **Exploit** will need ``cred_needed = 0`` and ``vul = None``
+
+- **Host** will have ``vul = []``, ``cred_tofind = 0`` and ``cred_needed = 0``
+
+
+These modifications are based on the Bachelor's Thesis of Norman Becker
+
+Contributions by Georg Blum @Class1G 
+
+ **Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN** Norman Becker et al. *2024*
+https://arxiv.org/abs/2407.15656
+
+-- Copyright © Norman Becker --
diff --git a/docs/source/tutorials/environment.rst b/docs/source/tutorials/environment.rst
diff --git a/docs/source/tutorials/example_network.png b/docs/source/tutorials/example_network.png
diff --git a/docs/source/tutorials/gym_load.rst b/docs/source/tutorials/gym_load.rst
diff --git a/docs/source/tutorials/index.rst b/docs/source/tutorials/index.rst
@@ -12,3 +12,6 @@ Tutorials
     environment
     scenarios
     creating_scenarios
+    credentials_vulnerabilities
+
+
diff --git a/docs/source/tutorials/installation.rst b/docs/source/tutorials/installation.rst
@@ -12,15 +12,16 @@ This framework is tested to work under Python 3.7 or later.
 The required dependencies:
 
 * Python >= 3.7
-* Gym >= 0.17
-* NumPy >= 1.18
-* PyYaml >= 5.3
+* Gym == 0.26.3
+* NumPy == 1.26.0
+* PyYaml >= 6.0.2
 
 For rendering:
 
-* NetworkX >= 2.4
-* prettytable >= 0.7.2
-* Matplotlib >= 3.1.3
+* NetworkX >= 3.3
+* prettytable >= 3.11.0
+* Matplotlib >= 3.9.2
+
 
 We recommend to use the bleeding-edge version and to install it by following the :ref:`dev-install`. If you want a simpler installation procedure and do not intend to modify yourself the learning algorithms etc., you can look at the :ref:`user-install`.
 

diff --git a/docs/source/tutorials/loading.rst b/docs/source/tutorials/loading.rst
diff --git a/docs/source/tutorials/scenarios.rst b/docs/source/tutorials/scenarios.rst
diff --git a/nasim/agents/__init__.py b/nasim/agents/__init__.py
diff --git a/nasim/agents/bruteforce_agent.py b/nasim/agents/bruteforce_agent.py
diff --git a/nasim/agents/dqn_agent.py b/nasim/agents/dqn_agent.py
diff --git a/nasim/agents/keyboard_agent.py b/nasim/agents/keyboard_agent.py
@@ -173,12 +173,13 @@ def run_keyboard_agent(env):
         print("\n" + LINE_BREAK2)
         print("OBSERVATION RECIEVED")
         print(LINE_BREAK2)
-        env.render()
+        print(env.current_state.tensor)
         print(f"Reward={r}")
         print(f"Done={done}")
         print(f"Step limit reached={step_limit_reached}")
         print(LINE_BREAK)
 
+
     return total_reward, total_steps, done
 
 

diff --git a/nasim/agents/policies/dqn_tiny.pt b/nasim/agents/policies/dqn_tiny.pt
diff --git a/nasim/agents/ql_agent.py b/nasim/agents/ql_agent.py
diff --git a/nasim/agents/ql_replay_agent.py b/nasim/agents/ql_replay_agent.py
diff --git a/nasim/agents/random_agent.py b/nasim/agents/random_agent.py
diff --git a/nasim/demo.py b/nasim/demo.py