Skip to content

Prerna_lab-hypothesis-testing-1 #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Prerna152 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Prerna_lab-hypothesis-testing-1 #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
279 changes: 279 additions & 0 deletions your-code/.ipynb_checkpoints/main-checkpoint.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,279 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Before your start:\n",
"- Read the README.md file\n",
"- Comment as much as you can and use the resources (README.md file)\n",
"- Happy learning!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# import numpy and pandas\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Challenge 1 - Exploring the Data\n",
"\n",
"In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Examine the `salaries` dataset using the `head` function below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are different departments in the city. List all departments and the count of employees in each department."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Challenge 2 - Hypothesis Tests\n",
"\n",
"In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n",
"\n",
"Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the `crosstab` function, find the department that has the most hourly workers. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Challenge 3: To practice - Constructing Confidence Intervals\n",
"\n",
"While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n",
").\n",
"\n",
"To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n",
"\n",
"\n",
"In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n",
"\n",
"The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n",
"\n",
"To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now construct the 95% confidence interval for all salaried employeed in the police in the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Bonus Challenge - Hypothesis Tests of Proportions\n",
"\n",
"Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n",
"\n",
"You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n",
"\n",
"In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your code here:\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading