From 0afaadf7b0ed1f03190cbf10ae4b29bb485cf8ce Mon Sep 17 00:00:00 2001 From: Prerna152 Date: Tue, 3 Sep 2024 17:34:34 +0200 Subject: [PATCH] update --- .../.ipynb_checkpoints/main-checkpoint.ipynb | 279 ++++++ your-code/main.ipynb | 912 ++++++++++++------ 2 files changed, 912 insertions(+), 279 deletions(-) create mode 100644 your-code/.ipynb_checkpoints/main-checkpoint.ipynb diff --git a/your-code/.ipynb_checkpoints/main-checkpoint.ipynb b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb new file mode 100644 index 0000000..59b955a --- /dev/null +++ b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb @@ -0,0 +1,279 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Before your start:\n", + "- Read the README.md file\n", + "- Comment as much as you can and use the resources (README.md file)\n", + "- Happy learning!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# import numpy and pandas\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 1 - Exploring the Data\n", + "\n", + "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Examine the `salaries` dataset using the `head` function below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are different departments in the city. List all departments and the count of employees in each department." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 2 - Hypothesis Tests\n", + "\n", + "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n", + "\n", + "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the `crosstab` function, find the department that has the most hourly workers. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 3: To practice - Constructing Confidence Intervals\n", + "\n", + "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n", + ").\n", + "\n", + "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n", + "\n", + "\n", + "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n", + "\n", + "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n", + "\n", + "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Bonus Challenge - Hypothesis Tests of Proportions\n", + "\n", + "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n", + "\n", + "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n", + "\n", + "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/your-code/main.ipynb b/your-code/main.ipynb index b2b6f8d..3874022 100644 --- a/your-code/main.ipynb +++ b/your-code/main.ipynb @@ -1,279 +1,633 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Before your start:\n", - "- Read the README.md file\n", - "- Comment as much as you can and use the resources (README.md file)\n", - "- Happy learning!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# import numpy and pandas\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Challenge 1 - Exploring the Data\n", - "\n", - "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Examine the `salaries` dataset using the `head` function below." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There are different departments in the city. List all departments and the count of employees in each department." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Challenge 2 - Hypothesis Tests\n", - "\n", - "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n", - "\n", - "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using the `crosstab` function, find the department that has the most hourly workers. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Challenge 3: To practice - Constructing Confidence Intervals\n", - "\n", - "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n", - ").\n", - "\n", - "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n", - "\n", - "\n", - "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n", - "\n", - "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n", - "\n", - "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Bonus Challenge - Hypothesis Tests of Proportions\n", - "\n", - "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n", - "\n", - "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n", - "\n", - "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n", - "\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Before your start:\n", + "- Read the README.md file\n", + "- Comment as much as you can and use the resources (README.md file)\n", + "- Happy learning!" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# import numpy and pandas\n", + "\n", + "import numpy as np\n", + "import pandas as pd" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 1 - Exploring the Data\n", + "\n", + "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Name Job Titles \\\n", + "0 AARON, JEFFERY M SERGEANT \n", + "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n", + "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n", + "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n", + "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n", + "\n", + " Department Full or Part-Time Salary or Hourly Typical Hours \\\n", + "0 POLICE F Salary NaN \n", + "1 POLICE F Salary NaN \n", + "2 GENERAL SERVICES F Salary NaN \n", + "3 WATER MGMNT F Salary NaN \n", + "4 OEMC P Hourly 20.0 \n", + "\n", + " Annual Salary Hourly Rate \n", + "0 101442.0 NaN \n", + "1 94122.0 NaN \n", + "2 101592.0 NaN \n", + "3 110064.0 NaN \n", + "4 NaN 19.86 \n", + "\n", + "RangeIndex: 33183 entries, 0 to 33182\n", + "Data columns (total 8 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 Name 33183 non-null object \n", + " 1 Job Titles 33183 non-null object \n", + " 2 Department 33183 non-null object \n", + " 3 Full or Part-Time 33183 non-null object \n", + " 4 Salary or Hourly 33183 non-null object \n", + " 5 Typical Hours 8022 non-null float64\n", + " 6 Annual Salary 25161 non-null float64\n", + " 7 Hourly Rate 8022 non-null float64\n", + "dtypes: float64(3), object(5)\n", + "memory usage: 2.0+ MB\n", + "None\n", + " Typical Hours Annual Salary Hourly Rate\n", + "count 8022.000000 25161.000000 8022.000000\n", + "mean 34.507604 86786.999790 32.788558\n", + "std 9.252077 21041.354602 12.112573\n", + "min 10.000000 7200.000000 2.650000\n", + "25% 20.000000 76266.000000 21.200000\n", + "50% 40.000000 90024.000000 35.600000\n", + "75% 40.000000 96060.000000 40.200000\n", + "max 40.000000 300000.000000 109.000000\n", + "Index(['Name', 'Job Titles', 'Department', 'Full or Part-Time',\n", + " 'Salary or Hourly', 'Typical Hours', 'Annual Salary', 'Hourly Rate'],\n", + " dtype='object')\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Load the dataset\n", + "data_set = 'Current_Employee_Names__Salaries__and_Position_Titles.csv'\n", + "df = pd.read_csv(data_set)\n", + "\n", + "# Examine the first few rows of the dataset\n", + "print(df.head())\n", + "\n", + "# Get an overview of the dataset, including column names and data types\n", + "print(df.info())\n", + "\n", + "# Check summary statistics for numerical columns\n", + "print(df.describe())\n", + "\n", + "# Display the column names to understand what data is available\n", + "print(df.columns)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Examine the `salaries` dataset using the `head` function below." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Name Job Titles \\\n", + "0 AARON, JEFFERY M SERGEANT \n", + "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n", + "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n", + "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n", + "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n", + "\n", + " Department Full or Part-Time Salary or Hourly Typical Hours \\\n", + "0 POLICE F Salary NaN \n", + "1 POLICE F Salary NaN \n", + "2 GENERAL SERVICES F Salary NaN \n", + "3 WATER MGMNT F Salary NaN \n", + "4 OEMC P Hourly 20.0 \n", + "\n", + " Annual Salary Hourly Rate \n", + "0 101442.0 NaN \n", + "1 94122.0 NaN \n", + "2 101592.0 NaN \n", + "3 110064.0 NaN \n", + "4 NaN 19.86 \n" + ] + } + ], + "source": [ + "# Your code here:\n", + "# Examine the first few rows of the dataset using the head function\n", + "print(df.head())\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Name 0\n", + "Job Titles 0\n", + "Department 0\n", + "Full or Part-Time 0\n", + "Salary or Hourly 0\n", + "Typical Hours 25161\n", + "Annual Salary 8022\n", + "Hourly Rate 25161\n", + "dtype: int64\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "# Check for missing data in each column\n", + "missing_data = df.isnull().sum()\n", + "\n", + "# Display the number of missing values for each column\n", + "print(missing_data)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "['Salary' 'Hourly']\n", + "Salary or Hourly\n", + "Salary 25161\n", + "Hourly 8022\n", + "Name: count, dtype: int64\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Check the unique values in the column that could indicate hourly vs salaried\n", + "print(df['Salary or Hourly'].unique())\n", + "\n", + "# Count the number of hourly vs. salaried employees\n", + "pay_count = df['Salary or Hourly'].value_counts()\n", + "\n", + "# Display the count\n", + "print(pay_count)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are different departments in the city. List all departments and the count of employees in each department." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Department\n", + "POLICE 13414\n", + "FIRE 4641\n", + "STREETS & SAN 2198\n", + "OEMC 2102\n", + "WATER MGMNT 1879\n", + "AVIATION 1629\n", + "TRANSPORTN 1140\n", + "PUBLIC LIBRARY 1015\n", + "GENERAL SERVICES 980\n", + "FAMILY & SUPPORT 615\n", + "FINANCE 560\n", + "HEALTH 488\n", + "CITY COUNCIL 411\n", + "LAW 407\n", + "BUILDINGS 269\n", + "COMMUNITY DEVELOPMENT 207\n", + "BUSINESS AFFAIRS 171\n", + "COPA 116\n", + "BOARD OF ELECTION 107\n", + "DoIT 99\n", + "PROCUREMENT 92\n", + "INSPECTOR GEN 87\n", + "MAYOR'S OFFICE 85\n", + "CITY CLERK 84\n", + "ANIMAL CONTRL 81\n", + "HUMAN RESOURCES 79\n", + "CULTURAL AFFAIRS 65\n", + "BUDGET & MGMT 46\n", + "ADMIN HEARNG 39\n", + "DISABILITIES 28\n", + "TREASURER 22\n", + "HUMAN RELATIONS 16\n", + "BOARD OF ETHICS 8\n", + "POLICE BOARD 2\n", + "LICENSE APPL COMM 1\n", + "Name: count, dtype: int64\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "department_counts = df['Department'].value_counts()\n", + "\n", + "# Display the list of departments and their respective employee counts\n", + "print(department_counts)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 2 - Hypothesis Tests\n", + "\n", + "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Test Statistic: 20.6198057854942\n", + "P-value: 4.3230240486229894e-92\n", + "Reject the null hypothesis: The hourly wage is significantly different from $30/hr.\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "from scipy.stats import ttest_1samp\n", + "\n", + "if 'Hourly Rate' in df.columns:\n", + " hourly_workers = df.dropna(subset=['Hourly Rate'])\n", + " \n", + " # Extract hourly wages\n", + " hourly_wages = hourly_workers['Hourly Rate']\n", + "\n", + " # Perform a one-sample t-test\n", + " test_statistic, p_value = ttest_1samp(hourly_wages, 30)\n", + "\n", + " # Print the results\n", + " print(f'Test Statistic: {test_statistic}')\n", + " print(f'P-value: {p_value}')\n", + "\n", + " # Interpretation at 95% confidence level\n", + " alpha = 0.05\n", + " if p_value < alpha:\n", + " print(\"Reject the null hypothesis: The hourly wage is significantly different from $30/hr.\")\n", + " else:\n", + " print(\"Fail to reject the null hypothesis: There is no significant difference from $30/hr.\")\n", + "else:\n", + " print(\"The column 'Hourly Rate' does not exist in the dataset.\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n", + "\n", + "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Test Statistic: 3.081859442309168\n", + "One-Tailed P-value: 0.0010306457005664076\n", + "Reject the null hypothesis: Salaries this year are significantly higher than $86,000/year.\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Filter salaried employees in the police department\n", + "# Assuming 'Department' column contains department names and 'Annual Salary' contains salaries\n", + "police_salaries = df[(df['Department'].str.contains('POLICE', case=False)) & df['Annual Salary'].notna()]\n", + "\n", + "# Extract salary data as a list or array\n", + "salaries = police_salaries['Annual Salary']\n", + "\n", + "# Perform a one-sample t-test\n", + "test_statistic, p_value_two_tailed = ttest_1samp(salaries, 86000)\n", + "\n", + "# Adjust the p-value for one-tailed test\n", + "p_value_one_tailed = p_value_two_tailed / 2\n", + "\n", + "# Print the results\n", + "print(f'Test Statistic: {test_statistic}')\n", + "print(f'One-Tailed P-value: {p_value_one_tailed}')\n", + "\n", + "# Interpretation at 95% confidence level (alpha = 0.05)\n", + "alpha = 0.05\n", + "\n", + "if (p_value_one_tailed < alpha) and (test_statistic > 0):\n", + " print(\"Reject the null hypothesis: Salaries this year are significantly higher than $86,000/year.\")\n", + "else:\n", + " print(\"Fail to reject the null hypothesis: No significant evidence that salaries are higher than $86,000/year.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the `crosstab` function, find the department that has the most hourly workers. " + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "col_0 Count\n", + "Department \n", + "STREETS & SAN 1862\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Identify hourly workers - Assuming 'Hourly Rate' column indicates hourly workers\n", + "hourly_workers = df[df['Hourly Rate'].notna()]\n", + "\n", + "# Use crosstab to count hourly workers by department\n", + "hourly_counts = pd.crosstab(index=hourly_workers['Department'], columns='Count')\n", + "\n", + "# Find the department with the most hourly workers\n", + "most_hourly_workers = hourly_counts.sort_values(by='Count', ascending=False).head(1)\n", + "\n", + "# Display the department with the most hourly workers\n", + "print(most_hourly_workers)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Reject the null hypothesis: The average hourly wage is less than $35/hour.\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Find the department with the most hourly workers\n", + "top_department = hourly_counts.sort_values(by='Count', ascending=False).index[0]\n", + "\n", + "# Extract hourly wages for the top department\n", + "top_department_wages = hourly_workers[hourly_workers['Department'] == top_department]['Hourly Rate']\n", + "\n", + "# Perform a one-sample t-test\n", + "t_stat, p_value = ttest_1samp(top_department_wages, 35)\n", + "\n", + "# Since this is a one-sided test, divide the p-value by 2 and check if the t-statistic is negative\n", + "p_value_one_sided = p_value / 2\n", + "\n", + "# Check if the test statistic supports the hypothesis that mean wage < $35\n", + "if (t_stat < 0) and (p_value_one_sided < 0.05):\n", + " print(\"Reject the null hypothesis: The average hourly wage is less than $35/hour.\")\n", + "else:\n", + " print(\"Fail to reject the null hypothesis: Not enough evidence to conclude the average hourly wage is less than $35/hour.\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 3: To practice - Constructing Confidence Intervals\n", + "\n", + "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n", + ").\n", + "\n", + "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n", + "\n", + "\n", + "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n", + "\n", + "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n", + "\n", + "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "95% Confidence Interval for the mean hourly wage: (32.52345834488425, 33.05365708767623)\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "from scipy.stats import t, sem\n", + "\n", + "# Calculate the sample mean of the hourly rates\n", + "mean_hourly_wage = hourly_workers['Hourly Rate'].mean()\n", + "\n", + "# Calculate the standard error of the mean (SEM)\n", + "standard_error = sem(hourly_workers['Hourly Rate'])\n", + "\n", + "# Define the confidence level and degrees of freedom\n", + "confidence_level = 0.95\n", + "degrees_of_freedom = len(hourly_workers) - 1\n", + "\n", + "# Compute the 95% confidence interval\n", + "confidence_interval = t.interval(confidence_level, degrees_of_freedom, loc=mean_hourly_wage, scale=standard_error)\n", + "\n", + "print(f\"95% Confidence Interval for the mean hourly wage: {confidence_interval}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "95% Confidence Interval for the mean salary of police salaried employees: (86177.05631531784, 86795.77269094894)\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "\n", + "# Filter the dataset for salaried employees in the police department\n", + "police_salaried = df[(df['Department'] == 'POLICE') & (df['Annual Salary'].notna())]\n", + "\n", + "# Calculate the mean salary for police salaried employees\n", + "mean_salary = police_salaried['Annual Salary'].mean()\n", + "\n", + "# Calculate the standard error of the mean (SEM)\n", + "standard_error = sem(police_salaried['Annual Salary'])\n", + "\n", + "# Define the confidence level and degrees of freedom\n", + "confidence_level = 0.95\n", + "degrees_of_freedom = len(police_salaried) - 1\n", + "\n", + "# Compute the 95% confidence interval\n", + "confidence_interval = t.interval(confidence_level, degrees_of_freedom, loc=mean_salary, scale=standard_error)\n", + "\n", + "print(f\"95% Confidence Interval for the mean salary of police salaried employees: {confidence_interval}\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Bonus Challenge - Hypothesis Tests of Proportions\n", + "\n", + "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n", + "\n", + "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n", + "\n", + "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here:\n", + "\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}