diff --git a/Open Notebook.onetoc2 b/Open Notebook.onetoc2
new file mode 100644
index 0000000..f1ce2f8
Binary files /dev/null and b/Open Notebook.onetoc2 differ
diff --git a/your-code/.ipynb_checkpoints/Open Notebook.onetoc2 b/your-code/.ipynb_checkpoints/Open Notebook.onetoc2
new file mode 100644
index 0000000..388db7f
Binary files /dev/null and b/your-code/.ipynb_checkpoints/Open Notebook.onetoc2 differ
diff --git a/your-code/.ipynb_checkpoints/main-checkpoint.ipynb b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb
new file mode 100644
index 0000000..66bd339
--- /dev/null
+++ b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb
@@ -0,0 +1,1411 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Before your start:\n",
+ "- Read the README.md file\n",
+ "- Comment as much as you can and use the resources (README.md file)\n",
+ "- Happy learning!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import numpy and pandas\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as st\n",
+ "import matplotlib.pyplot as plt\n",
+ "import seaborn as sns"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 1 - Exploring the Data\n",
+ "\n",
+ "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "data = pd.read_csv('Current_Employee_Names__Salaries__and_Position_Titles.csv')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Examine the `salaries` dataset using the `head` function below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " Name | \n",
+ " Job Titles | \n",
+ " Department | \n",
+ " Full or Part-Time | \n",
+ " Salary or Hourly | \n",
+ " Typical Hours | \n",
+ " Annual Salary | \n",
+ " Hourly Rate | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " AARON, JEFFERY M | \n",
+ " SERGEANT | \n",
+ " POLICE | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 101442.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " AARON, KARINA | \n",
+ " POLICE OFFICER (ASSIGNED AS DETECTIVE) | \n",
+ " POLICE | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 94122.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " AARON, KIMBERLEI R | \n",
+ " CHIEF CONTRACT EXPEDITER | \n",
+ " GENERAL SERVICES | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 101592.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " ABAD JR, VICENTE M | \n",
+ " CIVIL ENGINEER IV | \n",
+ " WATER MGMNT | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 110064.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " ABASCAL, REECE E | \n",
+ " TRAFFIC CONTROL AIDE-HOURLY | \n",
+ " OEMC | \n",
+ " P | \n",
+ " Hourly | \n",
+ " 20.0 | \n",
+ " NaN | \n",
+ " 19.86 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Name Job Titles \\\n",
+ "0 AARON, JEFFERY M SERGEANT \n",
+ "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n",
+ "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n",
+ "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n",
+ "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n",
+ "\n",
+ " Department Full or Part-Time Salary or Hourly Typical Hours \\\n",
+ "0 POLICE F Salary NaN \n",
+ "1 POLICE F Salary NaN \n",
+ "2 GENERAL SERVICES F Salary NaN \n",
+ "3 WATER MGMNT F Salary NaN \n",
+ "4 OEMC P Hourly 20.0 \n",
+ "\n",
+ " Annual Salary Hourly Rate \n",
+ "0 101442.0 NaN \n",
+ "1 94122.0 NaN \n",
+ "2 101592.0 NaN \n",
+ "3 110064.0 NaN \n",
+ "4 NaN 19.86 "
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "data.head()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Name 0\n",
+ "Job Titles 0\n",
+ "Department 0\n",
+ "Full or Part-Time 0\n",
+ "Salary or Hourly 0\n",
+ "Typical Hours 25161\n",
+ "Annual Salary 8022\n",
+ "Hourly Rate 25161\n",
+ "dtype: int64"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "data.isnull().sum()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Salary 25161\n",
+ "Hourly 8022\n",
+ "Name: Salary or Hourly, dtype: int64"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "value_counts = data['Salary or Hourly'].value_counts()\n",
+ "value_counts"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "There are different departments in the city. List all departments and the count of employees in each department."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " Name | \n",
+ "
\n",
+ " \n",
+ " | Department | \n",
+ " | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | ADMIN HEARNG | \n",
+ " 39 | \n",
+ "
\n",
+ " \n",
+ " | ANIMAL CONTRL | \n",
+ " 81 | \n",
+ "
\n",
+ " \n",
+ " | AVIATION | \n",
+ " 1629 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ELECTION | \n",
+ " 107 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ETHICS | \n",
+ " 8 | \n",
+ "
\n",
+ " \n",
+ " | BUDGET & MGMT | \n",
+ " 46 | \n",
+ "
\n",
+ " \n",
+ " | BUILDINGS | \n",
+ " 269 | \n",
+ "
\n",
+ " \n",
+ " | BUSINESS AFFAIRS | \n",
+ " 171 | \n",
+ "
\n",
+ " \n",
+ " | CITY CLERK | \n",
+ " 84 | \n",
+ "
\n",
+ " \n",
+ " | CITY COUNCIL | \n",
+ " 411 | \n",
+ "
\n",
+ " \n",
+ " | COMMUNITY DEVELOPMENT | \n",
+ " 207 | \n",
+ "
\n",
+ " \n",
+ " | COPA | \n",
+ " 116 | \n",
+ "
\n",
+ " \n",
+ " | CULTURAL AFFAIRS | \n",
+ " 65 | \n",
+ "
\n",
+ " \n",
+ " | DISABILITIES | \n",
+ " 28 | \n",
+ "
\n",
+ " \n",
+ " | DoIT | \n",
+ " 99 | \n",
+ "
\n",
+ " \n",
+ " | FAMILY & SUPPORT | \n",
+ " 615 | \n",
+ "
\n",
+ " \n",
+ " | FINANCE | \n",
+ " 560 | \n",
+ "
\n",
+ " \n",
+ " | FIRE | \n",
+ " 4641 | \n",
+ "
\n",
+ " \n",
+ " | GENERAL SERVICES | \n",
+ " 980 | \n",
+ "
\n",
+ " \n",
+ " | HEALTH | \n",
+ " 488 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RELATIONS | \n",
+ " 16 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RESOURCES | \n",
+ " 79 | \n",
+ "
\n",
+ " \n",
+ " | INSPECTOR GEN | \n",
+ " 87 | \n",
+ "
\n",
+ " \n",
+ " | LAW | \n",
+ " 407 | \n",
+ "
\n",
+ " \n",
+ " | LICENSE APPL COMM | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | MAYOR'S OFFICE | \n",
+ " 85 | \n",
+ "
\n",
+ " \n",
+ " | OEMC | \n",
+ " 2102 | \n",
+ "
\n",
+ " \n",
+ " | POLICE | \n",
+ " 13414 | \n",
+ "
\n",
+ " \n",
+ " | POLICE BOARD | \n",
+ " 2 | \n",
+ "
\n",
+ " \n",
+ " | PROCUREMENT | \n",
+ " 92 | \n",
+ "
\n",
+ " \n",
+ " | PUBLIC LIBRARY | \n",
+ " 1015 | \n",
+ "
\n",
+ " \n",
+ " | STREETS & SAN | \n",
+ " 2198 | \n",
+ "
\n",
+ " \n",
+ " | TRANSPORTN | \n",
+ " 1140 | \n",
+ "
\n",
+ " \n",
+ " | TREASURER | \n",
+ " 22 | \n",
+ "
\n",
+ " \n",
+ " | WATER MGMNT | \n",
+ " 1879 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Name\n",
+ "Department \n",
+ "ADMIN HEARNG 39\n",
+ "ANIMAL CONTRL 81\n",
+ "AVIATION 1629\n",
+ "BOARD OF ELECTION 107\n",
+ "BOARD OF ETHICS 8\n",
+ "BUDGET & MGMT 46\n",
+ "BUILDINGS 269\n",
+ "BUSINESS AFFAIRS 171\n",
+ "CITY CLERK 84\n",
+ "CITY COUNCIL 411\n",
+ "COMMUNITY DEVELOPMENT 207\n",
+ "COPA 116\n",
+ "CULTURAL AFFAIRS 65\n",
+ "DISABILITIES 28\n",
+ "DoIT 99\n",
+ "FAMILY & SUPPORT 615\n",
+ "FINANCE 560\n",
+ "FIRE 4641\n",
+ "GENERAL SERVICES 980\n",
+ "HEALTH 488\n",
+ "HUMAN RELATIONS 16\n",
+ "HUMAN RESOURCES 79\n",
+ "INSPECTOR GEN 87\n",
+ "LAW 407\n",
+ "LICENSE APPL COMM 1\n",
+ "MAYOR'S OFFICE 85\n",
+ "OEMC 2102\n",
+ "POLICE 13414\n",
+ "POLICE BOARD 2\n",
+ "PROCUREMENT 92\n",
+ "PUBLIC LIBRARY 1015\n",
+ "STREETS & SAN 2198\n",
+ "TRANSPORTN 1140\n",
+ "TREASURER 22\n",
+ "WATER MGMNT 1879"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "data.groupby(['Department'], dropna = False).aggregate({'Name':'count'})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 2 - Hypothesis Tests\n",
+ "\n",
+ "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "from scipy.stats import ttest_1samp"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#A commonly used rule of thumb suggests having a minimum sample size of 30. \n",
+ "#This guideline is often applied when the population distribution is assumed to be approximately normal, and the sample is intended for basic inferential statistics \n",
+ "\n",
+ "sample_1 = data[data['Salary or Hourly']== 'Hourly']['Hourly Rate'].sample(30)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "12421 19.86\n",
+ "25372 35.60\n",
+ "29743 12.49\n",
+ "9066 28.14\n",
+ "1024 44.55\n",
+ "4722 20.12\n",
+ "15179 36.21\n",
+ "20349 48.85\n",
+ "25897 48.90\n",
+ "26449 21.98\n",
+ "23362 48.90\n",
+ "7095 14.51\n",
+ "31492 50.25\n",
+ "14841 40.20\n",
+ "24391 40.20\n",
+ "5689 13.15\n",
+ "5122 34.57\n",
+ "26621 23.31\n",
+ "26850 28.14\n",
+ "15029 35.60\n",
+ "3350 17.68\n",
+ "13646 49.10\n",
+ "9594 19.38\n",
+ "15354 19.86\n",
+ "9627 45.07\n",
+ "15636 35.60\n",
+ "7230 34.57\n",
+ "17540 49.10\n",
+ "32081 35.60\n",
+ "18045 35.60\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#H0: \\mu_1 == 30\n",
+ "#H1 \\mu_1 != 30"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=1.2975300587942509, pvalue=0.20467590626852106, df=29)"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 30\n",
+ "st.ttest_1samp(sample_1,h0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.025"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "alpha = 0.05 #for a 95% two sided confidence interval\n",
+ "alpha_per_tail = alpha / 2\n",
+ "alpha_per_tail"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#P is 0.20, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n",
+ "\n",
+ "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_2 = data[data['Salary or Hourly']== 'Salary']['Annual Salary'].sample(30)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "21070 87006.0\n",
+ "12605 100776.0\n",
+ "3668 93354.0\n",
+ "13048 90024.0\n",
+ "20410 90024.0\n",
+ "5423 77484.0\n",
+ "1859 111474.0\n",
+ "1798 124728.0\n",
+ "13470 90024.0\n",
+ "12478 48078.0\n",
+ "1255 48078.0\n",
+ "17813 91272.0\n",
+ "2337 95484.0\n",
+ "14986 165000.0\n",
+ "9542 49104.0\n",
+ "20693 90024.0\n",
+ "21437 92274.0\n",
+ "20964 52176.0\n",
+ "17148 96060.0\n",
+ "3267 107988.0\n",
+ "27197 106596.0\n",
+ "1037 93354.0\n",
+ "29553 91080.0\n",
+ "32471 92274.0\n",
+ "24362 97386.0\n",
+ "15878 112854.0\n",
+ "1477 66948.0\n",
+ "18845 52320.0\n",
+ "20860 68616.0\n",
+ "9496 123894.0\n",
+ "Name: Annual Salary, dtype: float64"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=0.8963592270145543, pvalue=0.18872086850281916, df=29)"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 86000\n",
+ "st.ttest_1samp(sample_2,h0,alternative='greater')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "alpha = 0.05 #for a 95% one sided confidence interval\n",
+ "#P is 0.19, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "raw",
+ "metadata": {},
+ "source": [
+ "Using the `crosstab` function, find the department that has the most hourly workers. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Salary or Hourly | \n",
+ " Hourly | \n",
+ " Salary | \n",
+ "
\n",
+ " \n",
+ " | Department | \n",
+ " | \n",
+ " | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | STREETS & SAN | \n",
+ " 1862 | \n",
+ " 336 | \n",
+ "
\n",
+ " \n",
+ " | WATER MGMNT | \n",
+ " 1513 | \n",
+ " 366 | \n",
+ "
\n",
+ " \n",
+ " | OEMC | \n",
+ " 1273 | \n",
+ " 829 | \n",
+ "
\n",
+ " \n",
+ " | AVIATION | \n",
+ " 1082 | \n",
+ " 547 | \n",
+ "
\n",
+ " \n",
+ " | GENERAL SERVICES | \n",
+ " 765 | \n",
+ " 215 | \n",
+ "
\n",
+ " \n",
+ " | TRANSPORTN | \n",
+ " 725 | \n",
+ " 415 | \n",
+ "
\n",
+ " \n",
+ " | PUBLIC LIBRARY | \n",
+ " 299 | \n",
+ " 716 | \n",
+ "
\n",
+ " \n",
+ " | FAMILY & SUPPORT | \n",
+ " 287 | \n",
+ " 328 | \n",
+ "
\n",
+ " \n",
+ " | CITY COUNCIL | \n",
+ " 64 | \n",
+ " 347 | \n",
+ "
\n",
+ " \n",
+ " | FINANCE | \n",
+ " 44 | \n",
+ " 516 | \n",
+ "
\n",
+ " \n",
+ " | LAW | \n",
+ " 40 | \n",
+ " 367 | \n",
+ "
\n",
+ " \n",
+ " | ANIMAL CONTRL | \n",
+ " 19 | \n",
+ " 62 | \n",
+ "
\n",
+ " \n",
+ " | POLICE | \n",
+ " 10 | \n",
+ " 13404 | \n",
+ "
\n",
+ " \n",
+ " | MAYOR'S OFFICE | \n",
+ " 8 | \n",
+ " 77 | \n",
+ "
\n",
+ " \n",
+ " | BUSINESS AFFAIRS | \n",
+ " 7 | \n",
+ " 164 | \n",
+ "
\n",
+ " \n",
+ " | CULTURAL AFFAIRS | \n",
+ " 7 | \n",
+ " 58 | \n",
+ "
\n",
+ " \n",
+ " | COMMUNITY DEVELOPMENT | \n",
+ " 4 | \n",
+ " 203 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RESOURCES | \n",
+ " 4 | \n",
+ " 75 | \n",
+ "
\n",
+ " \n",
+ " | HEALTH | \n",
+ " 3 | \n",
+ " 485 | \n",
+ "
\n",
+ " \n",
+ " | PROCUREMENT | \n",
+ " 2 | \n",
+ " 90 | \n",
+ "
\n",
+ " \n",
+ " | FIRE | \n",
+ " 2 | \n",
+ " 4639 | \n",
+ "
\n",
+ " \n",
+ " | BUDGET & MGMT | \n",
+ " 2 | \n",
+ " 44 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RELATIONS | \n",
+ " 0 | \n",
+ " 16 | \n",
+ "
\n",
+ " \n",
+ " | INSPECTOR GEN | \n",
+ " 0 | \n",
+ " 87 | \n",
+ "
\n",
+ " \n",
+ " | DoIT | \n",
+ " 0 | \n",
+ " 99 | \n",
+ "
\n",
+ " \n",
+ " | LICENSE APPL COMM | \n",
+ " 0 | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | DISABILITIES | \n",
+ " 0 | \n",
+ " 28 | \n",
+ "
\n",
+ " \n",
+ " | COPA | \n",
+ " 0 | \n",
+ " 116 | \n",
+ "
\n",
+ " \n",
+ " | CITY CLERK | \n",
+ " 0 | \n",
+ " 84 | \n",
+ "
\n",
+ " \n",
+ " | POLICE BOARD | \n",
+ " 0 | \n",
+ " 2 | \n",
+ "
\n",
+ " \n",
+ " | BUILDINGS | \n",
+ " 0 | \n",
+ " 269 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ETHICS | \n",
+ " 0 | \n",
+ " 8 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ELECTION | \n",
+ " 0 | \n",
+ " 107 | \n",
+ "
\n",
+ " \n",
+ " | TREASURER | \n",
+ " 0 | \n",
+ " 22 | \n",
+ "
\n",
+ " \n",
+ " | ADMIN HEARNG | \n",
+ " 0 | \n",
+ " 39 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ "Salary or Hourly Hourly Salary\n",
+ "Department \n",
+ "STREETS & SAN 1862 336\n",
+ "WATER MGMNT 1513 366\n",
+ "OEMC 1273 829\n",
+ "AVIATION 1082 547\n",
+ "GENERAL SERVICES 765 215\n",
+ "TRANSPORTN 725 415\n",
+ "PUBLIC LIBRARY 299 716\n",
+ "FAMILY & SUPPORT 287 328\n",
+ "CITY COUNCIL 64 347\n",
+ "FINANCE 44 516\n",
+ "LAW 40 367\n",
+ "ANIMAL CONTRL 19 62\n",
+ "POLICE 10 13404\n",
+ "MAYOR'S OFFICE 8 77\n",
+ "BUSINESS AFFAIRS 7 164\n",
+ "CULTURAL AFFAIRS 7 58\n",
+ "COMMUNITY DEVELOPMENT 4 203\n",
+ "HUMAN RESOURCES 4 75\n",
+ "HEALTH 3 485\n",
+ "PROCUREMENT 2 90\n",
+ "FIRE 2 4639\n",
+ "BUDGET & MGMT 2 44\n",
+ "HUMAN RELATIONS 0 16\n",
+ "INSPECTOR GEN 0 87\n",
+ "DoIT 0 99\n",
+ "LICENSE APPL COMM 0 1\n",
+ "DISABILITIES 0 28\n",
+ "COPA 0 116\n",
+ "CITY CLERK 0 84\n",
+ "POLICE BOARD 0 2\n",
+ "BUILDINGS 0 269\n",
+ "BOARD OF ETHICS 0 8\n",
+ "BOARD OF ELECTION 0 107\n",
+ "TREASURER 0 22\n",
+ "ADMIN HEARNG 0 39"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "cross_tab = pd.crosstab(data['Department'], data['Salary or Hourly'])\n",
+ "cross_tab = cross_tab.sort_values(by='Hourly',ascending=False)\n",
+ "cross_tab"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "18084 36.21\n",
+ "31190 35.60\n",
+ "7715 35.60\n",
+ "8969 20.12\n",
+ "16330 36.13\n",
+ "3535 36.21\n",
+ "19796 36.21\n",
+ "23309 35.60\n",
+ "3123 20.77\n",
+ "26221 37.25\n",
+ "23725 44.55\n",
+ "8694 19.50\n",
+ "5898 20.12\n",
+ "19444 19.50\n",
+ "32133 35.60\n",
+ "25464 36.21\n",
+ "28953 37.56\n",
+ "29021 35.60\n",
+ "9732 37.56\n",
+ "9476 20.77\n",
+ "5087 36.21\n",
+ "3800 36.21\n",
+ "5148 35.60\n",
+ "31203 36.21\n",
+ "18640 36.21\n",
+ "1717 35.60\n",
+ "4616 35.60\n",
+ "9745 35.60\n",
+ "13372 36.21\n",
+ "9833 22.12\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_3 = data[(data['Department']== 'STREETS & SAN') & (data['Salary or Hourly']== 'Hourly')]['Hourly Rate'].sample(30)\n",
+ "sample_3"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=19.278801364117015, pvalue=1.0, df=29)"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 35\n",
+ "st.ttest_1samp(sample_2,h0,alternative='less')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "alpha = 0.05 #for a 95% one sided confidence interval\n",
+ "#P is 1, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 3: To practice - Constructing Confidence Intervals\n",
+ "\n",
+ "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n",
+ ").\n",
+ "\n",
+ "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n",
+ "\n",
+ "\n",
+ "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n",
+ "\n",
+ "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n",
+ "\n",
+ "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "21273 49.37\n",
+ "20275 35.60\n",
+ "30962 46.10\n",
+ "10651 25.10\n",
+ "9647 35.60\n",
+ "14763 15.65\n",
+ "399 44.25\n",
+ "5800 22.88\n",
+ "11256 25.10\n",
+ "32127 15.65\n",
+ "25545 40.20\n",
+ "26319 17.68\n",
+ "21741 46.10\n",
+ "13164 35.60\n",
+ "21631 8.25\n",
+ "20318 40.20\n",
+ "15932 40.20\n",
+ "19983 15.94\n",
+ "32207 19.86\n",
+ "20801 35.60\n",
+ "33145 35.57\n",
+ "26083 35.60\n",
+ "10071 36.21\n",
+ "10113 45.35\n",
+ "21791 20.88\n",
+ "17136 13.94\n",
+ "26985 45.07\n",
+ "20061 19.38\n",
+ "23043 2.65\n",
+ "24272 15.65\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_4 = data[data['Salary or Hourly']== 'Hourly']['Hourly Rate'].sample(30)\n",
+ "sample_4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Confidence Interval: (24.718752282526996, 34.29658105080636)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "\n",
+ "# Compute sample statistics\n",
+ "sample_mean = np.mean(sample_4)\n",
+ "sample_std = np.std(sample_4)\n",
+ "sample_size = len(sample_4)\n",
+ "\n",
+ "# Compute standard error\n",
+ "standard_error = sample_std / np.sqrt(sample_size)\n",
+ "\n",
+ "# Compute confidence interval\n",
+ "confidence_level = 0.95\n",
+ "degrees_of_freedom = sample_size - 1\n",
+ "confidence_interval = st.t.interval(confidence_level, degrees_of_freedom, loc=sample_mean, scale=standard_error)\n",
+ "\n",
+ "print(f\"Confidence Interval: {confidence_interval}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#we are 95% confident that the true value of the parameter lies within this range."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "32.78855771628128"
+ ]
+ },
+ "execution_count": 37,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "hourly_wages = (data[data['Salary or Hourly']== 'Hourly']['Hourly Rate']).mean()\n",
+ "hourly_wages"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#after testing it, we see the value is in fact in the range."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "7351 87006.0\n",
+ "6964 111252.0\n",
+ "23457 87006.0\n",
+ "15799 48078.0\n",
+ "24083 87006.0\n",
+ "6156 96060.0\n",
+ "27875 93354.0\n",
+ "24830 90024.0\n",
+ "18028 80016.0\n",
+ "11083 90024.0\n",
+ "4798 120252.0\n",
+ "1048 40392.0\n",
+ "18737 97440.0\n",
+ "21478 48078.0\n",
+ "31612 104628.0\n",
+ "5264 90024.0\n",
+ "20308 93354.0\n",
+ "29634 90024.0\n",
+ "12669 104628.0\n",
+ "15066 93354.0\n",
+ "3011 93354.0\n",
+ "10598 93354.0\n",
+ "2309 90024.0\n",
+ "2436 72510.0\n",
+ "7500 80016.0\n",
+ "10347 90024.0\n",
+ "32441 90024.0\n",
+ "30121 103932.0\n",
+ "16136 84054.0\n",
+ "3222 93354.0\n",
+ "Name: Annual Salary, dtype: float64"
+ ]
+ },
+ "execution_count": 39,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_5 = data[(data['Department']== 'POLICE') & (data['Salary or Hourly']== 'Salary')]['Annual Salary'].sample(30)\n",
+ "sample_5"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Confidence Interval: (81772.52497093234, 94403.87502906765)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Compute sample statistics\n",
+ "sample_mean = np.mean(sample_5)\n",
+ "sample_std = np.std(sample_5)\n",
+ "sample_size = len(sample_5)\n",
+ "\n",
+ "# Compute standard error\n",
+ "standard_error = sample_std / np.sqrt(sample_size)\n",
+ "\n",
+ "# Compute confidence interval\n",
+ "confidence_level = 0.95\n",
+ "degrees_of_freedom = sample_size - 1\n",
+ "confidence_interval = st.t.interval(confidence_level, degrees_of_freedom, loc=sample_mean, scale=standard_error)\n",
+ "\n",
+ "print(f\"Confidence Interval: {confidence_interval}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "86786.99979015153"
+ ]
+ },
+ "execution_count": 41,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "anual_salaries_police = (data[data['Salary or Hourly']== 'Salary']['Annual Salary']).mean()\n",
+ "anual_salaries_police"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#after testing it, we see the value is in fact in the range."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Bonus Challenge - Hypothesis Tests of Proportions\n",
+ "\n",
+ "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n",
+ "\n",
+ "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n",
+ "\n",
+ "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/your-code/Open Notebook.onetoc2 b/your-code/Open Notebook.onetoc2
new file mode 100644
index 0000000..8cdc2ee
Binary files /dev/null and b/your-code/Open Notebook.onetoc2 differ
diff --git a/your-code/main.ipynb b/your-code/main.ipynb
index b2b6f8d..66bd339 100644
--- a/your-code/main.ipynb
+++ b/your-code/main.ipynb
@@ -1,279 +1,1411 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Before your start:\n",
- "- Read the README.md file\n",
- "- Comment as much as you can and use the resources (README.md file)\n",
- "- Happy learning!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# import numpy and pandas\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Challenge 1 - Exploring the Data\n",
- "\n",
- "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Examine the `salaries` dataset using the `head` function below."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "There are different departments in the city. List all departments and the count of employees in each department."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Challenge 2 - Hypothesis Tests\n",
- "\n",
- "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n",
- "\n",
- "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Using the `crosstab` function, find the department that has the most hourly workers. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Challenge 3: To practice - Constructing Confidence Intervals\n",
- "\n",
- "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n",
- ").\n",
- "\n",
- "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n",
- "\n",
- "\n",
- "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n",
- "\n",
- "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n",
- "\n",
- "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Bonus Challenge - Hypothesis Tests of Proportions\n",
- "\n",
- "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n",
- "\n",
- "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n",
- "\n",
- "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Your code here:\n",
- "\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.3"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Before your start:\n",
+ "- Read the README.md file\n",
+ "- Comment as much as you can and use the resources (README.md file)\n",
+ "- Happy learning!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import numpy and pandas\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "import scipy.stats as st\n",
+ "import matplotlib.pyplot as plt\n",
+ "import seaborn as sns"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 1 - Exploring the Data\n",
+ "\n",
+ "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "data = pd.read_csv('Current_Employee_Names__Salaries__and_Position_Titles.csv')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Examine the `salaries` dataset using the `head` function below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " Name | \n",
+ " Job Titles | \n",
+ " Department | \n",
+ " Full or Part-Time | \n",
+ " Salary or Hourly | \n",
+ " Typical Hours | \n",
+ " Annual Salary | \n",
+ " Hourly Rate | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 0 | \n",
+ " AARON, JEFFERY M | \n",
+ " SERGEANT | \n",
+ " POLICE | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 101442.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 1 | \n",
+ " AARON, KARINA | \n",
+ " POLICE OFFICER (ASSIGNED AS DETECTIVE) | \n",
+ " POLICE | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 94122.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 2 | \n",
+ " AARON, KIMBERLEI R | \n",
+ " CHIEF CONTRACT EXPEDITER | \n",
+ " GENERAL SERVICES | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 101592.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 3 | \n",
+ " ABAD JR, VICENTE M | \n",
+ " CIVIL ENGINEER IV | \n",
+ " WATER MGMNT | \n",
+ " F | \n",
+ " Salary | \n",
+ " NaN | \n",
+ " 110064.0 | \n",
+ " NaN | \n",
+ "
\n",
+ " \n",
+ " | 4 | \n",
+ " ABASCAL, REECE E | \n",
+ " TRAFFIC CONTROL AIDE-HOURLY | \n",
+ " OEMC | \n",
+ " P | \n",
+ " Hourly | \n",
+ " 20.0 | \n",
+ " NaN | \n",
+ " 19.86 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Name Job Titles \\\n",
+ "0 AARON, JEFFERY M SERGEANT \n",
+ "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n",
+ "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n",
+ "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n",
+ "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n",
+ "\n",
+ " Department Full or Part-Time Salary or Hourly Typical Hours \\\n",
+ "0 POLICE F Salary NaN \n",
+ "1 POLICE F Salary NaN \n",
+ "2 GENERAL SERVICES F Salary NaN \n",
+ "3 WATER MGMNT F Salary NaN \n",
+ "4 OEMC P Hourly 20.0 \n",
+ "\n",
+ " Annual Salary Hourly Rate \n",
+ "0 101442.0 NaN \n",
+ "1 94122.0 NaN \n",
+ "2 101592.0 NaN \n",
+ "3 110064.0 NaN \n",
+ "4 NaN 19.86 "
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "data.head()\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Name 0\n",
+ "Job Titles 0\n",
+ "Department 0\n",
+ "Full or Part-Time 0\n",
+ "Salary or Hourly 0\n",
+ "Typical Hours 25161\n",
+ "Annual Salary 8022\n",
+ "Hourly Rate 25161\n",
+ "dtype: int64"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "data.isnull().sum()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Salary 25161\n",
+ "Hourly 8022\n",
+ "Name: Salary or Hourly, dtype: int64"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "value_counts = data['Salary or Hourly'].value_counts()\n",
+ "value_counts"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "There are different departments in the city. List all departments and the count of employees in each department."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | \n",
+ " Name | \n",
+ "
\n",
+ " \n",
+ " | Department | \n",
+ " | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | ADMIN HEARNG | \n",
+ " 39 | \n",
+ "
\n",
+ " \n",
+ " | ANIMAL CONTRL | \n",
+ " 81 | \n",
+ "
\n",
+ " \n",
+ " | AVIATION | \n",
+ " 1629 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ELECTION | \n",
+ " 107 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ETHICS | \n",
+ " 8 | \n",
+ "
\n",
+ " \n",
+ " | BUDGET & MGMT | \n",
+ " 46 | \n",
+ "
\n",
+ " \n",
+ " | BUILDINGS | \n",
+ " 269 | \n",
+ "
\n",
+ " \n",
+ " | BUSINESS AFFAIRS | \n",
+ " 171 | \n",
+ "
\n",
+ " \n",
+ " | CITY CLERK | \n",
+ " 84 | \n",
+ "
\n",
+ " \n",
+ " | CITY COUNCIL | \n",
+ " 411 | \n",
+ "
\n",
+ " \n",
+ " | COMMUNITY DEVELOPMENT | \n",
+ " 207 | \n",
+ "
\n",
+ " \n",
+ " | COPA | \n",
+ " 116 | \n",
+ "
\n",
+ " \n",
+ " | CULTURAL AFFAIRS | \n",
+ " 65 | \n",
+ "
\n",
+ " \n",
+ " | DISABILITIES | \n",
+ " 28 | \n",
+ "
\n",
+ " \n",
+ " | DoIT | \n",
+ " 99 | \n",
+ "
\n",
+ " \n",
+ " | FAMILY & SUPPORT | \n",
+ " 615 | \n",
+ "
\n",
+ " \n",
+ " | FINANCE | \n",
+ " 560 | \n",
+ "
\n",
+ " \n",
+ " | FIRE | \n",
+ " 4641 | \n",
+ "
\n",
+ " \n",
+ " | GENERAL SERVICES | \n",
+ " 980 | \n",
+ "
\n",
+ " \n",
+ " | HEALTH | \n",
+ " 488 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RELATIONS | \n",
+ " 16 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RESOURCES | \n",
+ " 79 | \n",
+ "
\n",
+ " \n",
+ " | INSPECTOR GEN | \n",
+ " 87 | \n",
+ "
\n",
+ " \n",
+ " | LAW | \n",
+ " 407 | \n",
+ "
\n",
+ " \n",
+ " | LICENSE APPL COMM | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | MAYOR'S OFFICE | \n",
+ " 85 | \n",
+ "
\n",
+ " \n",
+ " | OEMC | \n",
+ " 2102 | \n",
+ "
\n",
+ " \n",
+ " | POLICE | \n",
+ " 13414 | \n",
+ "
\n",
+ " \n",
+ " | POLICE BOARD | \n",
+ " 2 | \n",
+ "
\n",
+ " \n",
+ " | PROCUREMENT | \n",
+ " 92 | \n",
+ "
\n",
+ " \n",
+ " | PUBLIC LIBRARY | \n",
+ " 1015 | \n",
+ "
\n",
+ " \n",
+ " | STREETS & SAN | \n",
+ " 2198 | \n",
+ "
\n",
+ " \n",
+ " | TRANSPORTN | \n",
+ " 1140 | \n",
+ "
\n",
+ " \n",
+ " | TREASURER | \n",
+ " 22 | \n",
+ "
\n",
+ " \n",
+ " | WATER MGMNT | \n",
+ " 1879 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Name\n",
+ "Department \n",
+ "ADMIN HEARNG 39\n",
+ "ANIMAL CONTRL 81\n",
+ "AVIATION 1629\n",
+ "BOARD OF ELECTION 107\n",
+ "BOARD OF ETHICS 8\n",
+ "BUDGET & MGMT 46\n",
+ "BUILDINGS 269\n",
+ "BUSINESS AFFAIRS 171\n",
+ "CITY CLERK 84\n",
+ "CITY COUNCIL 411\n",
+ "COMMUNITY DEVELOPMENT 207\n",
+ "COPA 116\n",
+ "CULTURAL AFFAIRS 65\n",
+ "DISABILITIES 28\n",
+ "DoIT 99\n",
+ "FAMILY & SUPPORT 615\n",
+ "FINANCE 560\n",
+ "FIRE 4641\n",
+ "GENERAL SERVICES 980\n",
+ "HEALTH 488\n",
+ "HUMAN RELATIONS 16\n",
+ "HUMAN RESOURCES 79\n",
+ "INSPECTOR GEN 87\n",
+ "LAW 407\n",
+ "LICENSE APPL COMM 1\n",
+ "MAYOR'S OFFICE 85\n",
+ "OEMC 2102\n",
+ "POLICE 13414\n",
+ "POLICE BOARD 2\n",
+ "PROCUREMENT 92\n",
+ "PUBLIC LIBRARY 1015\n",
+ "STREETS & SAN 2198\n",
+ "TRANSPORTN 1140\n",
+ "TREASURER 22\n",
+ "WATER MGMNT 1879"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "data.groupby(['Department'], dropna = False).aggregate({'Name':'count'})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 2 - Hypothesis Tests\n",
+ "\n",
+ "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "from scipy.stats import ttest_1samp"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#A commonly used rule of thumb suggests having a minimum sample size of 30. \n",
+ "#This guideline is often applied when the population distribution is assumed to be approximately normal, and the sample is intended for basic inferential statistics \n",
+ "\n",
+ "sample_1 = data[data['Salary or Hourly']== 'Hourly']['Hourly Rate'].sample(30)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "12421 19.86\n",
+ "25372 35.60\n",
+ "29743 12.49\n",
+ "9066 28.14\n",
+ "1024 44.55\n",
+ "4722 20.12\n",
+ "15179 36.21\n",
+ "20349 48.85\n",
+ "25897 48.90\n",
+ "26449 21.98\n",
+ "23362 48.90\n",
+ "7095 14.51\n",
+ "31492 50.25\n",
+ "14841 40.20\n",
+ "24391 40.20\n",
+ "5689 13.15\n",
+ "5122 34.57\n",
+ "26621 23.31\n",
+ "26850 28.14\n",
+ "15029 35.60\n",
+ "3350 17.68\n",
+ "13646 49.10\n",
+ "9594 19.38\n",
+ "15354 19.86\n",
+ "9627 45.07\n",
+ "15636 35.60\n",
+ "7230 34.57\n",
+ "17540 49.10\n",
+ "32081 35.60\n",
+ "18045 35.60\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#H0: \\mu_1 == 30\n",
+ "#H1 \\mu_1 != 30"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=1.2975300587942509, pvalue=0.20467590626852106, df=29)"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 30\n",
+ "st.ttest_1samp(sample_1,h0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "0.025"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "alpha = 0.05 #for a 95% two sided confidence interval\n",
+ "alpha_per_tail = alpha / 2\n",
+ "alpha_per_tail"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#P is 0.20, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n",
+ "\n",
+ "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_2 = data[data['Salary or Hourly']== 'Salary']['Annual Salary'].sample(30)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "21070 87006.0\n",
+ "12605 100776.0\n",
+ "3668 93354.0\n",
+ "13048 90024.0\n",
+ "20410 90024.0\n",
+ "5423 77484.0\n",
+ "1859 111474.0\n",
+ "1798 124728.0\n",
+ "13470 90024.0\n",
+ "12478 48078.0\n",
+ "1255 48078.0\n",
+ "17813 91272.0\n",
+ "2337 95484.0\n",
+ "14986 165000.0\n",
+ "9542 49104.0\n",
+ "20693 90024.0\n",
+ "21437 92274.0\n",
+ "20964 52176.0\n",
+ "17148 96060.0\n",
+ "3267 107988.0\n",
+ "27197 106596.0\n",
+ "1037 93354.0\n",
+ "29553 91080.0\n",
+ "32471 92274.0\n",
+ "24362 97386.0\n",
+ "15878 112854.0\n",
+ "1477 66948.0\n",
+ "18845 52320.0\n",
+ "20860 68616.0\n",
+ "9496 123894.0\n",
+ "Name: Annual Salary, dtype: float64"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=0.8963592270145543, pvalue=0.18872086850281916, df=29)"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 86000\n",
+ "st.ttest_1samp(sample_2,h0,alternative='greater')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "alpha = 0.05 #for a 95% one sided confidence interval\n",
+ "#P is 0.19, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "raw",
+ "metadata": {},
+ "source": [
+ "Using the `crosstab` function, find the department that has the most hourly workers. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Salary or Hourly | \n",
+ " Hourly | \n",
+ " Salary | \n",
+ "
\n",
+ " \n",
+ " | Department | \n",
+ " | \n",
+ " | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | STREETS & SAN | \n",
+ " 1862 | \n",
+ " 336 | \n",
+ "
\n",
+ " \n",
+ " | WATER MGMNT | \n",
+ " 1513 | \n",
+ " 366 | \n",
+ "
\n",
+ " \n",
+ " | OEMC | \n",
+ " 1273 | \n",
+ " 829 | \n",
+ "
\n",
+ " \n",
+ " | AVIATION | \n",
+ " 1082 | \n",
+ " 547 | \n",
+ "
\n",
+ " \n",
+ " | GENERAL SERVICES | \n",
+ " 765 | \n",
+ " 215 | \n",
+ "
\n",
+ " \n",
+ " | TRANSPORTN | \n",
+ " 725 | \n",
+ " 415 | \n",
+ "
\n",
+ " \n",
+ " | PUBLIC LIBRARY | \n",
+ " 299 | \n",
+ " 716 | \n",
+ "
\n",
+ " \n",
+ " | FAMILY & SUPPORT | \n",
+ " 287 | \n",
+ " 328 | \n",
+ "
\n",
+ " \n",
+ " | CITY COUNCIL | \n",
+ " 64 | \n",
+ " 347 | \n",
+ "
\n",
+ " \n",
+ " | FINANCE | \n",
+ " 44 | \n",
+ " 516 | \n",
+ "
\n",
+ " \n",
+ " | LAW | \n",
+ " 40 | \n",
+ " 367 | \n",
+ "
\n",
+ " \n",
+ " | ANIMAL CONTRL | \n",
+ " 19 | \n",
+ " 62 | \n",
+ "
\n",
+ " \n",
+ " | POLICE | \n",
+ " 10 | \n",
+ " 13404 | \n",
+ "
\n",
+ " \n",
+ " | MAYOR'S OFFICE | \n",
+ " 8 | \n",
+ " 77 | \n",
+ "
\n",
+ " \n",
+ " | BUSINESS AFFAIRS | \n",
+ " 7 | \n",
+ " 164 | \n",
+ "
\n",
+ " \n",
+ " | CULTURAL AFFAIRS | \n",
+ " 7 | \n",
+ " 58 | \n",
+ "
\n",
+ " \n",
+ " | COMMUNITY DEVELOPMENT | \n",
+ " 4 | \n",
+ " 203 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RESOURCES | \n",
+ " 4 | \n",
+ " 75 | \n",
+ "
\n",
+ " \n",
+ " | HEALTH | \n",
+ " 3 | \n",
+ " 485 | \n",
+ "
\n",
+ " \n",
+ " | PROCUREMENT | \n",
+ " 2 | \n",
+ " 90 | \n",
+ "
\n",
+ " \n",
+ " | FIRE | \n",
+ " 2 | \n",
+ " 4639 | \n",
+ "
\n",
+ " \n",
+ " | BUDGET & MGMT | \n",
+ " 2 | \n",
+ " 44 | \n",
+ "
\n",
+ " \n",
+ " | HUMAN RELATIONS | \n",
+ " 0 | \n",
+ " 16 | \n",
+ "
\n",
+ " \n",
+ " | INSPECTOR GEN | \n",
+ " 0 | \n",
+ " 87 | \n",
+ "
\n",
+ " \n",
+ " | DoIT | \n",
+ " 0 | \n",
+ " 99 | \n",
+ "
\n",
+ " \n",
+ " | LICENSE APPL COMM | \n",
+ " 0 | \n",
+ " 1 | \n",
+ "
\n",
+ " \n",
+ " | DISABILITIES | \n",
+ " 0 | \n",
+ " 28 | \n",
+ "
\n",
+ " \n",
+ " | COPA | \n",
+ " 0 | \n",
+ " 116 | \n",
+ "
\n",
+ " \n",
+ " | CITY CLERK | \n",
+ " 0 | \n",
+ " 84 | \n",
+ "
\n",
+ " \n",
+ " | POLICE BOARD | \n",
+ " 0 | \n",
+ " 2 | \n",
+ "
\n",
+ " \n",
+ " | BUILDINGS | \n",
+ " 0 | \n",
+ " 269 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ETHICS | \n",
+ " 0 | \n",
+ " 8 | \n",
+ "
\n",
+ " \n",
+ " | BOARD OF ELECTION | \n",
+ " 0 | \n",
+ " 107 | \n",
+ "
\n",
+ " \n",
+ " | TREASURER | \n",
+ " 0 | \n",
+ " 22 | \n",
+ "
\n",
+ " \n",
+ " | ADMIN HEARNG | \n",
+ " 0 | \n",
+ " 39 | \n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ "Salary or Hourly Hourly Salary\n",
+ "Department \n",
+ "STREETS & SAN 1862 336\n",
+ "WATER MGMNT 1513 366\n",
+ "OEMC 1273 829\n",
+ "AVIATION 1082 547\n",
+ "GENERAL SERVICES 765 215\n",
+ "TRANSPORTN 725 415\n",
+ "PUBLIC LIBRARY 299 716\n",
+ "FAMILY & SUPPORT 287 328\n",
+ "CITY COUNCIL 64 347\n",
+ "FINANCE 44 516\n",
+ "LAW 40 367\n",
+ "ANIMAL CONTRL 19 62\n",
+ "POLICE 10 13404\n",
+ "MAYOR'S OFFICE 8 77\n",
+ "BUSINESS AFFAIRS 7 164\n",
+ "CULTURAL AFFAIRS 7 58\n",
+ "COMMUNITY DEVELOPMENT 4 203\n",
+ "HUMAN RESOURCES 4 75\n",
+ "HEALTH 3 485\n",
+ "PROCUREMENT 2 90\n",
+ "FIRE 2 4639\n",
+ "BUDGET & MGMT 2 44\n",
+ "HUMAN RELATIONS 0 16\n",
+ "INSPECTOR GEN 0 87\n",
+ "DoIT 0 99\n",
+ "LICENSE APPL COMM 0 1\n",
+ "DISABILITIES 0 28\n",
+ "COPA 0 116\n",
+ "CITY CLERK 0 84\n",
+ "POLICE BOARD 0 2\n",
+ "BUILDINGS 0 269\n",
+ "BOARD OF ETHICS 0 8\n",
+ "BOARD OF ELECTION 0 107\n",
+ "TREASURER 0 22\n",
+ "ADMIN HEARNG 0 39"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "cross_tab = pd.crosstab(data['Department'], data['Salary or Hourly'])\n",
+ "cross_tab = cross_tab.sort_values(by='Hourly',ascending=False)\n",
+ "cross_tab"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "18084 36.21\n",
+ "31190 35.60\n",
+ "7715 35.60\n",
+ "8969 20.12\n",
+ "16330 36.13\n",
+ "3535 36.21\n",
+ "19796 36.21\n",
+ "23309 35.60\n",
+ "3123 20.77\n",
+ "26221 37.25\n",
+ "23725 44.55\n",
+ "8694 19.50\n",
+ "5898 20.12\n",
+ "19444 19.50\n",
+ "32133 35.60\n",
+ "25464 36.21\n",
+ "28953 37.56\n",
+ "29021 35.60\n",
+ "9732 37.56\n",
+ "9476 20.77\n",
+ "5087 36.21\n",
+ "3800 36.21\n",
+ "5148 35.60\n",
+ "31203 36.21\n",
+ "18640 36.21\n",
+ "1717 35.60\n",
+ "4616 35.60\n",
+ "9745 35.60\n",
+ "13372 36.21\n",
+ "9833 22.12\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 24,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_3 = data[(data['Department']== 'STREETS & SAN') & (data['Salary or Hourly']== 'Hourly')]['Hourly Rate'].sample(30)\n",
+ "sample_3"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "TtestResult(statistic=19.278801364117015, pvalue=1.0, df=29)"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "h0 = 35\n",
+ "st.ttest_1samp(sample_2,h0,alternative='less')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "alpha = 0.05 #for a 95% one sided confidence interval\n",
+ "#P is 1, Fail to reject null hyphotesis"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Challenge 3: To practice - Constructing Confidence Intervals\n",
+ "\n",
+ "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n",
+ ").\n",
+ "\n",
+ "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n",
+ "\n",
+ "\n",
+ "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n",
+ "\n",
+ "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n",
+ "\n",
+ "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "21273 49.37\n",
+ "20275 35.60\n",
+ "30962 46.10\n",
+ "10651 25.10\n",
+ "9647 35.60\n",
+ "14763 15.65\n",
+ "399 44.25\n",
+ "5800 22.88\n",
+ "11256 25.10\n",
+ "32127 15.65\n",
+ "25545 40.20\n",
+ "26319 17.68\n",
+ "21741 46.10\n",
+ "13164 35.60\n",
+ "21631 8.25\n",
+ "20318 40.20\n",
+ "15932 40.20\n",
+ "19983 15.94\n",
+ "32207 19.86\n",
+ "20801 35.60\n",
+ "33145 35.57\n",
+ "26083 35.60\n",
+ "10071 36.21\n",
+ "10113 45.35\n",
+ "21791 20.88\n",
+ "17136 13.94\n",
+ "26985 45.07\n",
+ "20061 19.38\n",
+ "23043 2.65\n",
+ "24272 15.65\n",
+ "Name: Hourly Rate, dtype: float64"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sample_4 = data[data['Salary or Hourly']== 'Hourly']['Hourly Rate'].sample(30)\n",
+ "sample_4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Confidence Interval: (24.718752282526996, 34.29658105080636)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "\n",
+ "# Compute sample statistics\n",
+ "sample_mean = np.mean(sample_4)\n",
+ "sample_std = np.std(sample_4)\n",
+ "sample_size = len(sample_4)\n",
+ "\n",
+ "# Compute standard error\n",
+ "standard_error = sample_std / np.sqrt(sample_size)\n",
+ "\n",
+ "# Compute confidence interval\n",
+ "confidence_level = 0.95\n",
+ "degrees_of_freedom = sample_size - 1\n",
+ "confidence_interval = st.t.interval(confidence_level, degrees_of_freedom, loc=sample_mean, scale=standard_error)\n",
+ "\n",
+ "print(f\"Confidence Interval: {confidence_interval}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#we are 95% confident that the true value of the parameter lies within this range."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "32.78855771628128"
+ ]
+ },
+ "execution_count": 37,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "hourly_wages = (data[data['Salary or Hourly']== 'Hourly']['Hourly Rate']).mean()\n",
+ "hourly_wages"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#after testing it, we see the value is in fact in the range."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "7351 87006.0\n",
+ "6964 111252.0\n",
+ "23457 87006.0\n",
+ "15799 48078.0\n",
+ "24083 87006.0\n",
+ "6156 96060.0\n",
+ "27875 93354.0\n",
+ "24830 90024.0\n",
+ "18028 80016.0\n",
+ "11083 90024.0\n",
+ "4798 120252.0\n",
+ "1048 40392.0\n",
+ "18737 97440.0\n",
+ "21478 48078.0\n",
+ "31612 104628.0\n",
+ "5264 90024.0\n",
+ "20308 93354.0\n",
+ "29634 90024.0\n",
+ "12669 104628.0\n",
+ "15066 93354.0\n",
+ "3011 93354.0\n",
+ "10598 93354.0\n",
+ "2309 90024.0\n",
+ "2436 72510.0\n",
+ "7500 80016.0\n",
+ "10347 90024.0\n",
+ "32441 90024.0\n",
+ "30121 103932.0\n",
+ "16136 84054.0\n",
+ "3222 93354.0\n",
+ "Name: Annual Salary, dtype: float64"
+ ]
+ },
+ "execution_count": 39,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Your code here:\n",
+ "\n",
+ "sample_5 = data[(data['Department']== 'POLICE') & (data['Salary or Hourly']== 'Salary')]['Annual Salary'].sample(30)\n",
+ "sample_5"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Confidence Interval: (81772.52497093234, 94403.87502906765)\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Compute sample statistics\n",
+ "sample_mean = np.mean(sample_5)\n",
+ "sample_std = np.std(sample_5)\n",
+ "sample_size = len(sample_5)\n",
+ "\n",
+ "# Compute standard error\n",
+ "standard_error = sample_std / np.sqrt(sample_size)\n",
+ "\n",
+ "# Compute confidence interval\n",
+ "confidence_level = 0.95\n",
+ "degrees_of_freedom = sample_size - 1\n",
+ "confidence_interval = st.t.interval(confidence_level, degrees_of_freedom, loc=sample_mean, scale=standard_error)\n",
+ "\n",
+ "print(f\"Confidence Interval: {confidence_interval}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "86786.99979015153"
+ ]
+ },
+ "execution_count": 41,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "anual_salaries_police = (data[data['Salary or Hourly']== 'Salary']['Annual Salary']).mean()\n",
+ "anual_salaries_police"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#after testing it, we see the value is in fact in the range."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Bonus Challenge - Hypothesis Tests of Proportions\n",
+ "\n",
+ "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n",
+ "\n",
+ "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n",
+ "\n",
+ "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Your code here:\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}