diff --git a/your-code/lab_imbalance.ipynb b/your-code/lab_imbalance.ipynb
index dbb15e1..2e52bf9 100644
--- a/your-code/lab_imbalance.ipynb
+++ b/your-code/lab_imbalance.ipynb
@@ -1,147 +1,526 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Inbalanced Classes\n",
- "## In this lab, we are going to explore a case of imbalanced classes. \n",
- "\n",
- "\n",
- "Like we disussed in class, when we have noisy data, if we are not careful, we can end up fitting our model to the noise in the data and not the 'signal'-- the factors that actually determine the outcome. This is called overfitting, and results in good results in training, and in bad results when the model is applied to real data. Similarly, we could have a model that is too simplistic to accurately model the signal. This produces a model that doesnt work well (ever). \n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Note: before doing the first commit, make sure you don't include the large csv file, either by adding it to .gitignore, or by deleting it."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### First, download the data from: https://www.kaggle.com/datasets/chitwanmanchanda/fraudulent-transactions-data?resource=download . Import the dataset and provide some discriptive statistics and plots. What do you think will be the important features in determining the outcome?\n",
- "### Note: don't use the entire dataset, use a sample instead, with n=100000 elements, so your computer doesn't freeze."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### What is the distribution of the outcome? "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your response here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Clean the dataset. Pre-process it to make it suitable for ML training. Feel free to explore, drop, encode, transform, etc. Whatever you feel will improve the model score."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Run a logisitc regression classifier and evaluate its accuracy."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Now pick a model of your choice and evaluate its accuracy."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Which model worked better and how do you know?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your response here"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Note: before doing the first commit, make sure you don't include the large csv file, either by adding it to .gitignore, or by deleting it."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.8"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Inbalanced Classes\n",
+ "## In this lab, we are going to explore a case of imbalanced classes. \n",
+ "\n",
+ "\n",
+ "Like we disussed in class, when we have noisy data, if we are not careful, we can end up fitting our model to the noise in the data and not the 'signal'-- the factors that actually determine the outcome. This is called overfitting, and results in good results in training, and in bad results when the model is applied to real data. Similarly, we could have a model that is too simplistic to accurately model the signal. This produces a model that doesnt work well (ever). \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Note: before doing the first commit, make sure you don't include the large csv file, either by adding it to .gitignore, or by deleting it."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### First, download the data from: https://www.kaggle.com/datasets/chitwanmanchanda/fraudulent-transactions-data?resource=download . Import the dataset and provide some discriptive statistics and plots. What do you think will be the important features in determining the outcome?\n",
+ "### Note: don't use the entire dataset, use a sample instead, with n=100000 elements, so your computer doesn't freeze."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "import pandas as pd\n",
+ "import numpy as no\n",
+ "import matplotlib.pyplot as plt\n",
+ "import seaborn as sns\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.model_selection import train_test_split\n",
+ "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
+ "from sklearn.compose import ColumnTransformer\n",
+ "from sklearn.pipeline import Pipeline\n",
+ "from sklearn.ensemble import RandomForestClassifier\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ " step type amount nameOrig oldbalanceOrg newbalanceOrig \\\n",
+ "6322570 688 CASH_IN 23557.12 C867750533 8059.00 31616.12 \n",
+ "3621196 274 PAYMENT 6236.13 C601099070 0.00 0.00 \n",
+ "1226256 133 PAYMENT 33981.87 C279540931 18745.72 0.00 \n",
+ "2803274 225 CASH_OUT 263006.42 C11675531 20072.00 0.00 \n",
+ "3201247 249 CASH_OUT 152013.74 C530649214 20765.00 0.00 \n",
+ "\n",
+ " nameDest oldbalanceDest newbalanceDest isFraud isFlaggedFraud \n",
+ "6322570 C1026934669 169508.66 145951.53 0 0 \n",
+ "3621196 M701283411 0.00 0.00 0 0 \n",
+ "1226256 M577905776 0.00 0.00 0 0 \n",
+ "2803274 C529577791 390253.56 653259.98 0 0 \n",
+ "3201247 C1304175579 252719.19 404732.93 0 0 \n"
+ ]
+ }
+ ],
+ "source": [
+ "\n",
+ "samples = 100000\n",
+ "fraud = pd.read_csv('Fraud.csv').sample(n=samples, random_state=1)\n",
+ "print(fraud.head())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " step \n",
+ " amount \n",
+ " oldbalanceOrg \n",
+ " newbalanceOrig \n",
+ " oldbalanceDest \n",
+ " newbalanceDest \n",
+ " isFraud \n",
+ " isFlaggedFraud \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " count \n",
+ " 100000.000000 \n",
+ " 1.000000e+05 \n",
+ " 1.000000e+05 \n",
+ " 1.000000e+05 \n",
+ " 1.000000e+05 \n",
+ " 1.000000e+05 \n",
+ " 100000.000000 \n",
+ " 100000.0 \n",
+ " \n",
+ " \n",
+ " mean \n",
+ " 243.229430 \n",
+ " 1.815534e+05 \n",
+ " 8.283566e+05 \n",
+ " 8.498642e+05 \n",
+ " 1.102745e+06 \n",
+ " 1.230714e+06 \n",
+ " 0.001240 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " std \n",
+ " 142.711863 \n",
+ " 6.168152e+05 \n",
+ " 2.875843e+06 \n",
+ " 2.912484e+06 \n",
+ " 3.531001e+06 \n",
+ " 3.836813e+06 \n",
+ " 0.035192 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " min \n",
+ " 1.000000 \n",
+ " 1.800000e-01 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " 25% \n",
+ " 155.000000 \n",
+ " 1.344656e+04 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000e+00 \n",
+ " 0.000000 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " 50% \n",
+ " 239.000000 \n",
+ " 7.550210e+04 \n",
+ " 1.466800e+04 \n",
+ " 0.000000e+00 \n",
+ " 1.315374e+05 \n",
+ " 2.163863e+05 \n",
+ " 0.000000 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " 75% \n",
+ " 335.000000 \n",
+ " 2.092775e+05 \n",
+ " 1.070410e+05 \n",
+ " 1.449728e+05 \n",
+ " 9.458267e+05 \n",
+ " 1.115200e+06 \n",
+ " 0.000000 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ " max \n",
+ " 742.000000 \n",
+ " 4.669816e+07 \n",
+ " 4.381886e+07 \n",
+ " 4.368662e+07 \n",
+ " 3.281945e+08 \n",
+ " 3.279981e+08 \n",
+ " 1.000000 \n",
+ " 0.0 \n",
+ " \n",
+ " \n",
+ "
\n",
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# Plot distribution of 'isFraud'\n",
+ "\n",
+ "plt.figure(figsize=(6, 4))\n",
+ "sns.countplot(data=fraud, x='isFraud')\n",
+ "plt.title('Distribution of Fraudulent Transactions')\n",
+ "plt.xlabel('Is Fraudulent?')\n",
+ "plt.ylabel('Count')\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### What is the distribution of the outcome? "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\"\"\"The majority of the outcome is zeor meaning that they are not fraudulent.\"\"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Clean the dataset. Pre-process it to make it suitable for ML training. Feel free to explore, drop, encode, transform, etc. Whatever you feel will improve the model score."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Index: 100000 entries, 6322570 to 4679267\n",
+ "Data columns (total 11 columns):\n",
+ " # Column Non-Null Count Dtype \n",
+ "--- ------ -------------- ----- \n",
+ " 0 step 100000 non-null int64 \n",
+ " 1 type 100000 non-null object \n",
+ " 2 amount 100000 non-null float64\n",
+ " 3 nameOrig 100000 non-null object \n",
+ " 4 oldbalanceOrg 100000 non-null float64\n",
+ " 5 newbalanceOrig 100000 non-null float64\n",
+ " 6 nameDest 100000 non-null object \n",
+ " 7 oldbalanceDest 100000 non-null float64\n",
+ " 8 newbalanceDest 100000 non-null float64\n",
+ " 9 isFraud 100000 non-null int64 \n",
+ " 10 isFlaggedFraud 100000 non-null int64 \n",
+ "dtypes: float64(5), int64(3), object(3)\n",
+ "memory usage: 9.2+ MB\n"
+ ]
+ }
+ ],
+ "source": [
+ "fraud.info()\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "fraud = pd.get_dummies(fraud, columns=['type'], drop_first=True)\n",
+ "\n",
+ "fraud.drop(columns=['nameOrig', 'nameDest'], inplace=True)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "fraud.drop(columns=['step', 'isFlaggedFraud'], inplace=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "amount 0\n",
+ "oldbalanceOrg 0\n",
+ "newbalanceOrig 0\n",
+ "oldbalanceDest 0\n",
+ "newbalanceDest 0\n",
+ "isFraud 0\n",
+ "type_CASH_OUT 0\n",
+ "type_DEBIT 0\n",
+ "type_PAYMENT 0\n",
+ "type_TRANSFER 0\n",
+ "dtype: int64\n"
+ ]
+ }
+ ],
+ "source": [
+ "missing_values = fraud.isnull().sum()\n",
+ "print(missing_values)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "X = fraud.drop(columns=['isFraud'])\n",
+ "y = fraud['isFraud']\n",
+ "\n",
+ "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
+ "\n",
+ "numeric_features = X.select_dtypes(include=['int64', 'float64']).columns.tolist()\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "preprocessor = ColumnTransformer(\n",
+ " transformers=[\n",
+ " ('num', StandardScaler(), numeric_features)\n",
+ " ])\n",
+ "\n",
+ "pipeline = Pipeline(steps=[('preprocessor', preprocessor)])\n",
+ "\n",
+ "X_train_preprocessed = pipeline.fit_transform(X_train)\n",
+ "X_test_preprocessed = pipeline.transform(X_test)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Run a logisitc regression classifier and evaluate its accuracy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Accuracy: 0.99925\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.linear_model import LogisticRegression\n",
+ "from sklearn.metrics import accuracy_score\n",
+ "\n",
+ "model = LogisticRegression()\n",
+ "\n",
+ "model.fit(X_train_preprocessed, y_train)\n",
+ "y_pred = model.predict(X_test_preprocessed)\n",
+ "\n",
+ "accuracy = accuracy_score(y_test, y_pred)\n",
+ "print(\"Accuracy:\", accuracy)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Now pick a model of your choice and evaluate its accuracy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Random Forest Classifier Accuracy: 0.99945\n"
+ ]
+ }
+ ],
+ "source": [
+ "from sklearn.ensemble import RandomForestClassifier\n",
+ "from sklearn.metrics import accuracy_score\n",
+ "\n",
+ "rf_classifier = RandomForestClassifier(n_estimators=100, random_state=42)\n",
+ "rf_classifier.fit(X_train_preprocessed, y_train)\n",
+ "\n",
+ "y_pred_rf = rf_classifier.predict(X_test_preprocessed)\n",
+ "\n",
+ "accuracy_rf = accuracy_score(y_test, y_pred_rf)\n",
+ "print(\"Random Forest Classifier Accuracy:\", accuracy_rf)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Which model worked better and how do you know?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ " Random Forest Classifier achieved a slightly higher accuracy score of 0.99945"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Note: before doing the first commit, make sure you don't include the large csv file, either by adding it to .gitignore, or by deleting it."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}