Fuzzing, A/B and Canary Testing

ABTesting.md

@@ -0,0 +1,134 @@
+# A/B Testing on doodle quarkus
+
+## Introduction
+A/B testing is a process in which we provide different versions of an application to the end user to see how it performs by retrieving different metrics.
+For this tutorial, we will show how to use GrowthBook to perform an A/B testing on the front-end of the doodle application.
+
+## Setting up growthbook
+The first thing to do is to set up the growthBook application, the easiest way is to self-host it by using a docker-compose file provided by growthbook.
+```yaml
+# docker-compose.yaml
+version: '3'
+services:
+  mongo:
+    image: "mongo:latest"
+    environment:
+      - MONGO_INITDB_ROOT_USERNAME=root
+      - MONGO_INITDB_ROOT_PASSWORD=password
+  growthbook:
+    image: "growthbook/growthbook:latest"
+    ports:
+      - "3000:3000"
+      - "3100:3100"
+    depends_on:
+      - mongo
+    environment:
+      - MONGODB_URI=mongodb://root:password@mongo:27017/
+    volumes:
+      - uploads:/usr/local/src/app/packages/back-end/uploads
+volumes:
+  uploads:
+```
+And then to lauch it with the command 
+```bash
+docker-compose up
+```
+This will download the mongodb and growthbook images and launch them.<br>
+You can then reach the self-hosted growthbook website by accessing [http://localhost:3000]()<br>
+
+You will first be prompted with the first page in which you will be able to create a account.<br>
+<img src='./Images/Account.png' width=650>
+
+
+## Setting up the doodle front-end
+### Lauch the app
+To simplify this tutorial we will only be running the front-end of the application, which can be launched by using these commands:<br>
+`npm install` and `npm start`<br>
+And then accessing the front-end on [http://localhost:4200]()
+### Integrate GrowthBook in the angular application
+Now that doodle and GrowthBook are set up, we can update our doodle app to use GrowthBook<br>
+To do so, first acces "Feature>All Features" on the left side of the GrowthBook website.<br>
+<img src='./Images/allFeatures.png' width=180><br>
+And then click on `View instructions`<br>
+<img src='./Images/installSDK.png' width=500><br>
+What you are seeing now is some explanation on how to use growthbook with your application, and it is what I used to implement it in the doodle app<br>
+Growthbook then provides starter code for a few different languages, but we are going to use the javascript code by putting it in an angular service.<br>
+The service that I have created is available [here](./front/src/app/abtesting.service.ts). Essentially, the service has two methods:<br>
+- An init method which initializes the growthbook service by creating a growthbook instance, then reaching the api endpoint and finally setting the attributes of the current user (We will talk more precisely about those later).
+- A getState method that we will use to retrieve wether a user has a feature activated.
+
+To use this service, the only thing to change is the line 5:
+```javascript
+const FEATURES_ENDPOINT = "_YOUR_SDK_ENDPOINT_";
+```
+And replacing "_YOUR_SDK_ENDPOINT_" by the endpoint in "Features>Environments>SDK endpoint".
+
+
+## Setting up a feature in growthbook
+
+As mentionned earlier, the purpose of A/B testing is to provide different version of the application to the end user. In order to do so, we use what is called "features". After setting up these features, we will get those check with angular wether those features are up or not.<br>
+We will now need to create our first feature by accessing "Features>All Features" and then clicking on "Add first feature"<br>
+<img src='./Images/firstFeature.png' width=500><br>
+You will then be prompted with a "create feature" tab in which you can define different setting about the feature you want to publish. For our app, we will only provide the users with a centered title. Note that the default value is set to "On", so without changing anything every user will have this feature enabled<br>
+<img src='./Images/FFeature.png' width=500><br>
+
+## Using the feature in angular
+
+As said earlier, we will center a title of the website by using the features of growthbook, to do so, we need to modify the code of the create-poll component a little<br>
+First we have to inject the abtesting service in the component by adding it in the providers and in the constructor:
+```javascript
+...
+    providers: [MessageService, PollService, FullCalendarComponent, ABTestingService]
+...
+    centerValue = false
+...
+    await this.ABTesting.init();
+
+    this.centerValue = this.ABTesting.getState("center-title")
+...
+```
+And then we can modify the component in the html to change the position of the title.
+```html
+    <img src="assets/flat_logo.png" alt="Logo Simba" height="50px" style="display: block" [ngStyle]="{ 'marginBottom': '1rem','margin-left':(centerValue)? 'auto':'','margin-right':(centerValue)? 'auto':'' }"/>
+
+```
+Since the center-title feature is available by default, the title is now centered when we are in the create-poll page.<br>
+<img src='./Images/Centered.png' width=700><br>
+We have created a feature available by default but It is not A/B testing yet, to archive this we will have to modify the feature.
+
+## Modifying the feature for A/B testing
+You can now access the feature you have created in "Features>All Features" and click on your feature to access its settings.<br>
+You can now scroll down to the "Override Rules" section, different rules can be added to this feature but we will select **A/B experiment**.
+<img src='./Images/Overide.png' width=700>
+
+### Targetting Attributes 
+The first setting that matters is the "assign value based on attribute". Every user has a set of attribute that is associated to them, I kept those provided by default but they can be changed in "Features>Attributes".<br>
+As you can see below, In the program I also kept those by default but in a real production environnement those attributes could be retrieved after a user identification.
+```javascript
+  public setattribute() {
+    this.growthbook.setAttributes({
+      "id": "foo",
+      "deviceId": "foo",
+      "company": "foo",
+      "loggedIn": true,
+      "employee": true,
+      "country": "foo",
+      "browser": "foo",
+      "url": "foo"
+    });
+  }
+
+```
+So for our A/B testing experiment, we will keep the id as the targetting attribute.
+
+### Exposure, Variation and Weights
+In the section just below the attribute selection, we have the section where we can select the percents of users who will be part of the experiment as well as the different splits and values. we will keep the default values of 50% for each split and then click save. Once its done you can now click "Review and publish" on the top of the page to release the test".
+
+## Testing the experiment
+Now, we have a 50% chance of getting either, a value of true or false for the center-title value. For example if we change the value of our attribute "id" from foo to something random like "test", if we get lucky, we get this page with the title on the left:<br>
+Note that growthbook uses deterministic hashing, so each id has a static value associated with it.<br>
+<img src='./Images/notCenter.png' width=700>
+
+## Analysis
+Growthbook also allows for analysis about the different experiments, to do that you can connect a datasource to it and define metrics that helps you compare the versions of your apps. For example for our centered title we could mesure the time on site or the bounce rate which would give us an insight on what the users think about the feature.
+

CanaryTesting.md

@@ -0,0 +1,47 @@
+# Canary Testing
+
+## Introduction
+
+Canary testing is a method of software verification whose principle is to deploy a new version of the application to a subset of users. It allows to test the application in a real situation while not affecting all the population using it, and in case of failure, affects less users.
+
+For example, Meta uses the Canary Testing method on its employess applications and then can gather information quickly by gathering data and feedback from the employees without affecting the external users.
+
+Canary testing and A/B testing are similar in deployment, but their objective is different. Canary testing is meant to test resilience of new features, while A/B testing is meant to gather feedback on user experience between multiple already working features.
+
+Like A/B testing, the Canary method fits well with continous software deployment.
+
+## Implementation
+
+Canary and A/B testing being similar, we can then use the same tutorial than for [A/B testing](./ABTesting.md).
+
+However, due to the difference in the objectives of both methods, we have to modify certain parameters. The first parameter is the population. We do not want to have a large part of the user base (for example 50% like in the A/B tutorial), so we can have different methods of user separation : 
+  - Employees separation
+  - 90/10 ratio (or less depending on the magnitude of the changes or risks, going as low as 99/1), 10 being the percentage of the population getting the new features to test.
+
+### Employees separation
+
+After the feature creation in your Growthbook, instead of selecting A/B Experiment, we will have to go to "Features>Attributes" and then click on "Edit Attributes" on the top right corner.
+
+![image](https://user-images.githubusercontent.com/101655310/204324115-24561660-38ec-40d7-9d89-27a4a8526173.png)
+
+You then need to tick the "Identifier" column corresponding to the "employee" attribute, like shown in the image above. This will allow you to select "employee" as a separation factor later.
+
+To do that, go to "Features>All Features" and select the feature created before. Like in the A/B testing tutorial, select **A/B Experiment** in the "Override Rules" section.
+
+Instead of keeping the id in "Assign value based on attribute", you are now able to select "employee". Tick "Customize Split" in the "Exposure, Variations and Weight" section and bring the slider bar to 100% for the "On" option. This activates the option "On" for every employee, but no other user.
+
+![image](https://user-images.githubusercontent.com/101655310/204326712-d9952354-b4dd-4a55-aea0-d648ed98629c.png)
+
+
+### 99/1
+
+If you want to test your application with a certain percentage of the user population, you then need to use "id" instead of "employee" in your override rule, and then bring the slider to 1% for the "On" option instead of 100%. Only 1% of users will then have the new features.
+
+![image](https://user-images.githubusercontent.com/101655310/204328006-da186631-397a-4324-9320-121c191f15c5.png)
+
+You are now able to change different separation rules based on your testing strategy !
+
+### Logging
+
+For the Canary testing method to work fully, you have to be able to detect errors within the population using the new features. For that, you need to be sure that your logging process is functionning well and easy for the users to use.
+If the test users are the employees, error reporting can even be easier because they will be able to quickly report and describe the errors.

Fuzzing.md

@@ -0,0 +1,122 @@
+# Fuzzing
+## Introduction
+Fuzzing is a way of testing applications by providing random data as input trying to trigger errors such as exceptions, memory leak, stack smashing, double free or even higer level flaws such as sql injection.
+
+Fuzzing can be applied to various type of systems : API, binaries, web apps..
+
+Fuzzing is primarily used in two fields : V&V and Security  
+Depending on the objectives the data generation techniques vary :  
+- Grammar
+- Dictionnary
+- random mutators
+
+## Fuzzing Doodle
+
+Doodle is an application with multiple surface attack.  
+We could choose to fuzz :
+- The API
+- The back-end binaries
+
+## Fuzz the API
+
+### Using Burp Suite  
+Burp Suite is a paid software with a free limited version. Its purpose is to do penetration testing of web application. The intruder feature enables us to fuzz our Doodle API.
+
+After downloading Burp, you can open it and create a new project or a temporary project :  
+<img src='./Images/createProject.png' width=650>  
+
+
+Choose default settings :  
+<img src='./Images/settings.png' width=650>  
+
+
+Go to the "Intruder" tab and set up the HTTP request and choose what variable you will fuzz :  
+<img src='./Images/intruderTabPosition.png' width=650>  
+
+
+Then in the Payload tab choose how to generate data :  
+<img src='./Images/intruderTabPayload.png' width=650>  
+
+
+You can now launch the "attack" and check the server response.
+You can use this tool to either check if your API works as intended, or, to try find bugs.  
+
+### Using a custom script (Python)  
+If no off the shelf software meet your needs, you can easily build custom script to fuzz your API  
+```python
+import requests
+import random
+import string
+
+NUMBER_OF_TRY = 100
+
+def get_random_string(length):
+    # choose from all lowercase letter
+    letters = string.hexdigits
+    result_str = ''.join(random.choice(letters) for i in range(length))
+    return result_str
+
+# Define the URL of the API you want to fuzz
+api_url = "http://localhost:4200/api/pool/slug/"
+
+
+# Define a list of input values to use for fuzzing the API
+fuzz_values = [get_random_string(24) for i in range(NUMBER_OF_TRY)]
+
+
+# Iterate over the list of fuzz values and make a request to the API for each value
+for val in fuzz_values:
+    # Make a GET request to the API using the current fuzz value as a query parameter
+    response = requests.get(api_url, params={"fuzz_param": val})
+
+    # Print the status code of the response
+    print(f"Response for fuzz value {val}: {response.status_code}")
+```  
+
+this script is calling the API trying to find hidden pool.
+
+```
+Response for fuzz value 538BEB050aAbEE2E788eB988: 404
+Response for fuzz value FCb2eCd5175F8E69fcd3dcF4: 404
+Response for fuzz value eA47ecbd2eBe57393Eb7b2D7: 404
+Response for fuzz value 2f0b85c6A83094bEa1b47aBa: 404
+Response for fuzz value 1f83332CD7fDBBFC74c7d596: 404
+Response for fuzz value 4F63a34CbeBDAdD89bE15cc2: 404
+Response for fuzz value 6D53D03a1F2Db4B330Afd72E: 404
+Response for fuzz value 57D7bdAb1FFE4a45eF7Ba7Cb: 404
+Response for fuzz value B73b0F2B6Bd3Cb844FD77C40: 404
+Response for fuzz value c03fEBaAd2E84F2FEDDbEA6F: 404
+Response for fuzz value 0eC0EA9Fd8F802DdbB4C32eF: 404
+Response for fuzz value 7cEC2eBA2c8CaA7e450Bf633: 404
+Response for fuzz value 4DB83db2ed15D5da820fCeA0: 404
+Response for fuzz value CA2C4999d1fE4eF654fbA763: 404
+Response for fuzz value 9fE25DfFDFaFf23f5b4269D8: 404
+```
+
+This is a fraction of the result we get. Since we did not find any hidden pool we receive 404 errors.  
+
+## Fuzz the backend binary 
+
+To fuzz our backend, we will use Jazzer. Jazzer is a fuzzer working with JVMs. The fuzzer will try to cover as much code as possible by using JVMs features to know what is the code coverage acheived by each input data.  
+The basic usage is to use the autofuzz fonctionality and let Jazzer decide what input and what mutators to use.  
+
+In the case of doodle we have not to forget to lauch the etherpad and database dockers.  
+
+```./jazzer --cp=./tlcdemoApp-1.0.0-SNAPSHOT.jar --autofuzz=org.apache.commons.imaging.Imaging::getBufferedImage```  
+
+if jazzer find a bug it will save the raw input in a file and also a java code to reproduce the crash and easily turn it into a junit test case.  
+<img src='./Images/crash.png' width=650>  
+
+And the generated java code :  
+```java
+public class Crash_adc83b19e793491b1c6ea0fd8b46cd9f32e592fc {
+  public static void main(String[] args) throws Throwable {
+    (((java.util.function.Supplier<net.sourceforge.plantuml.yaml.SimpleYamlParser>) (() -> {net.sourceforge.plantuml.yaml.SimpleYamlParser autofuzzVariable0 = new net.sourceforge.plantuml.yaml.SimpleYamlParser(); return autofuzzVariable0;})).get()).parse((java.util.List) null);
+  }
+}
+```  
+
+If needed you can check the documentation to create more fine grained fuzzing target and build your own data generator.  
+
+## Conclusion  
+Fuzzing is way to test or attack software or API. For an in depth view on this subject you can check the fuzzing book https://www.fuzzingbook.org/ 

Readme.md

@@ -0,0 +1,20 @@
+# Fuzzing, A/B and Canary Testing
+
+## Group Members
+ - Ewen Brune
+ - Charly Reux
+ - Matilin Thomas
+ - Léo Thuillier
+
+
+## Fuzzing
+
+Our tutorial on Fuzzing is available [here](./Fuzzing.md)
+
+## A/B Testing
+
+Our tutorial on A/B testing is available [here](./ABTesting.md)
+
+## Canary testing
+
+Our tutorial on Canary is available [here](./CanaryTesting.md)

api/.dockerignore

@@ -0,0 +1,5 @@
+*
+!target/*-runner
+!target/*-runner.jar
+!target/lib/*
+!target/quarkus-app/*

api/.gitignore

@@ -0,0 +1,37 @@
+#Maven
+target/
+pom.xml.tag
+pom.xml.releaseBackup
+pom.xml.versionsBackup
+release.properties
+.mvn/wrapper/maven-wrapper.jar
+# Eclipse
+.project
+.classpath
+.settings/
+bin/
+
+# IntelliJ
+.idea
+*.ipr
+*.iml
+*.iws
+
+# NetBeans
+nb-configuration.xml
+
+# Visual Studio Code
+.vscode
+.factorypath
+
+# OSX
+.DS_Store
+
+# Vim
+*.swp
+*.swo
+
+# patch
+*.orig
+*.rej
+

api/.mvn/wrapper/maven-wrapper.properties

@@ -0,0 +1,2 @@
+distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.zip
+wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.6/maven-wrapper-0.5.6.jar

api/APIKEY.txt

@@ -0,0 +1 @@
+19d89ca52bc0fa4f19d6325464d9d7a032649b9fa68c111514627081e2784b4a