diff --git a/your-code/challenge-1.ipynb b/your-code/challenge-1.ipynb index b184a33..6a1ad41 100644 --- a/your-code/challenge-1.ipynb +++ b/your-code/challenge-1.ipynb @@ -1,247 +1,544 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Challenge 1 - T-test\n", - "\n", - "In statistics, t-test is used to test if two data samples have a significant difference between their means. There are two types of t-test:\n", - "\n", - "* **Student's t-test** (a.k.a. independent or uncorrelated t-test). This type of t-test is to compare the samples of **two independent populations** (e.g. test scores of students in two different classes). `scipy` provides the [`ttest_ind`](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.ttest_ind.html) method to conduct student's t-test.\n", - "\n", - "* **Paired t-test** (a.k.a. dependent or correlated t-test). This type of t-test is to compare the samples of **the same population** (e.g. scores of different tests of students in the same class). `scipy` provides the [`ttest_re`](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.ttest_rel.html) method to conduct paired t-test.\n", - "\n", - "Both types of t-tests return a number which is called the **p-value**. If p-value is below 0.05, we can confidently declare the null-hypothesis is rejected and the difference is significant. If p-value is between 0.05 and 0.1, we may also declare the null-hypothesis is rejected but we are not highly confident. If p-value is above 0.1 we do not reject the null-hypothesis.\n", - "\n", - "Read more about the t-test in [this article](http://b.link/test50) and [this Quora](http://b.link/unpaired97). Make sure you understand when to use which type of t-test. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Import libraries\n", - "import pandas as pd" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Import dataset\n", - "\n", - "In this challenge we will work on the Pokemon dataset. The goal is to test whether different groups of pokemon (e.g. Legendary vs Normal, Generation 1 vs 2, single-type vs dual-type) have different stats (e.g. HP, Attack, Defense, etc.)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here:\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### First we want to define a function with which we can test the means of a feature set of two samples. \n", - "\n", - "In the next cell you'll see the annotations of the Python function that explains what this function does and its arguments and returned value. This type of annotation is called **docstring** which is a convention used among Python developers. The docstring convention allows developers to write consistent tech documentations for their codes so that others can read. It also allows some websites to automatically parse the docstrings and display user-friendly documentations.\n", - "\n", - "Follow the specifications of the docstring and complete the function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def t_test_features(s1, s2, features=['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed', 'Total']):\n", - " \"\"\"Test means of a feature set of two samples\n", - " \n", - " Args:\n", - " s1 (dataframe): sample 1\n", - " s2 (dataframe): sample 2\n", - " features (list): an array of features to test\n", - " \n", - " Returns:\n", - " dict: a dictionary of t-test scores for each feature where the feature name is the key and the p-value is the value\n", - " \"\"\"\n", - " results = {}\n", - "\n", - " # Your code here\n", - " \n", - " return results" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Using the `t_test_features` function, conduct t-test for Lengendary vs non-Legendary pokemons.\n", - "\n", - "*Hint: your output should look like below:*\n", - "\n", - "```\n", - "{'HP': 1.0026911708035284e-13,\n", - " 'Attack': 2.520372449236646e-16,\n", - " 'Defense': 4.8269984949193316e-11,\n", - " 'Sp. Atk': 1.5514614112239812e-21,\n", - " 'Sp. Def': 2.2949327864052826e-15,\n", - " 'Speed': 1.049016311882451e-18,\n", - " 'Total': 9.357954335957446e-47}\n", - " ```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### From the test results above, what conclusion can you make? Do Legendary and non-Legendary pokemons have significantly different stats on each feature?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your comment here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Next, conduct t-test for Generation 1 and Generation 2 pokemons." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### What conclusions can you make?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your comment here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Compare pokemons who have single type vs those having two types." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### What conclusions can you make?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your comment here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Now, we want to compare whether there are significant differences of `Attack` vs `Defense` and `Sp. Atk` vs `Sp. Def` of all pokemons. Please write your code below.\n", - "\n", - "*Hint: are you comparing different populations or the same population?*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### What conclusions can you make?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your comment here" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 1 - T-test\n", + "\n", + "In statistics, t-test is used to test if two data samples have a significant difference between their means. There are two types of t-test:\n", + "\n", + "* **Student's t-test** (a.k.a. independent or uncorrelated t-test). This type of t-test is to compare the samples of **two independent populations** (e.g. test scores of students in two different classes). `scipy` provides the [`ttest_ind`](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.ttest_ind.html) method to conduct student's t-test.\n", + "\n", + "* **Paired t-test** (a.k.a. dependent or correlated t-test). This type of t-test is to compare the samples of **the same population** (e.g. scores of different tests of students in the same class). `scipy` provides the [`ttest_re`](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.ttest_rel.html) method to conduct paired t-test.\n", + "\n", + "Both types of t-tests return a number which is called the **p-value**. If p-value is below 0.05, we can confidently declare the null-hypothesis is rejected and the difference is significant. If p-value is between 0.05 and 0.1, we may also declare the null-hypothesis is rejected but we are not highly confident. If p-value is above 0.1 we do not reject the null-hypothesis.\n", + "\n", + "Read more about the t-test in [this article](http://b.link/test50) and [this Quora](http://b.link/unpaired97). Make sure you understand when to use which type of t-test. " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Import libraries\n", + "import pandas as pd\n", + "import scipy.stats as st" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Import dataset\n", + "\n", + "In this challenge we will work on the Pokemon dataset. The goal is to test whether different groups of pokemon (e.g. Legendary vs Normal, Generation 1 vs 2, single-type vs dual-type) have different stats (e.g. HP, Attack, Defense, etc.)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
#NameType 1Type 2TotalHPAttackDefenseSp. AtkSp. DefSpeedGenerationLegendary
01BulbasaurGrassPoison3184549496565451False
12IvysaurGrassPoison4056062638080601False
23VenusaurGrassPoison525808283100100801False
33VenusaurMega VenusaurGrassPoison62580100123122120801False
44CharmanderFireNaN3093952436050651False
\n", + "
" + ], + "text/plain": [ + " # Name Type 1 Type 2 Total HP Attack Defense \\\n", + "0 1 Bulbasaur Grass Poison 318 45 49 49 \n", + "1 2 Ivysaur Grass Poison 405 60 62 63 \n", + "2 3 Venusaur Grass Poison 525 80 82 83 \n", + "3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 \n", + "4 4 Charmander Fire NaN 309 39 52 43 \n", + "\n", + " Sp. Atk Sp. Def Speed Generation Legendary \n", + "0 65 65 45 1 False \n", + "1 80 80 60 1 False \n", + "2 100 100 80 1 False \n", + "3 122 120 80 1 False \n", + "4 60 50 65 1 False " + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "RangeIndex: 800 entries, 0 to 799\n", + "Data columns (total 13 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 # 800 non-null int64 \n", + " 1 Name 800 non-null object\n", + " 2 Type 1 800 non-null object\n", + " 3 Type 2 414 non-null object\n", + " 4 Total 800 non-null int64 \n", + " 5 HP 800 non-null int64 \n", + " 6 Attack 800 non-null int64 \n", + " 7 Defense 800 non-null int64 \n", + " 8 Sp. Atk 800 non-null int64 \n", + " 9 Sp. Def 800 non-null int64 \n", + " 10 Speed 800 non-null int64 \n", + " 11 Generation 800 non-null int64 \n", + " 12 Legendary 800 non-null bool \n", + "dtypes: bool(1), int64(9), object(3)\n", + "memory usage: 75.9+ KB\n" + ] + }, + { + "data": { + "text/plain": [ + "None" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Your code here:\n", + "pokemon = pd.read_csv('Pokemon.csv')\n", + "display(pokemon.head())\n", + "display(pokemon.info())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### First we want to define a function with which we can test the means of a feature set of two samples. \n", + "\n", + "In the next cell you'll see the annotations of the Python function that explains what this function does and its arguments and returned value. This type of annotation is called **docstring** which is a convention used among Python developers. The docstring convention allows developers to write consistent tech documentations for their codes so that others can read. It also allows some websites to automatically parse the docstrings and display user-friendly documentations.\n", + "\n", + "Follow the specifications of the docstring and complete the function." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [], + "source": [ + "def t_test_features(s1, s2, features=['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed', 'Total']):\n", + " \"\"\"Test means of a feature set of two samples\n", + " \n", + " Args:\n", + " s1 (dataframe): sample 1\n", + " s2 (dataframe): sample 2\n", + " features (list): an array of features to test\n", + " \n", + " Returns:\n", + " dict: a dictionary of t-test scores for each feature where the feature name is the key and the p-value is the value\n", + " \"\"\"\n", + " results = {}\n", + "\n", + " # Your code here\n", + " for feature in features:\n", + " statistics, p_value = st.ttest_ind(s1[feature], s2[feature], equal_var=False)\n", + " if feature not in results.keys():\n", + " results[feature]= p_value\n", + " else:\n", + " pass\n", + "\n", + " return results" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Using the `t_test_features` function, conduct t-test for Lengendary vs non-Legendary pokemons.\n", + "\n", + "*Hint: your output should look like below:*\n", + "\n", + "```\n", + "{'HP': 1.0026911708035284e-13,\n", + " 'Attack': 2.520372449236646e-16,\n", + " 'Defense': 4.8269984949193316e-11,\n", + " 'Sp. Atk': 1.5514614112239812e-21,\n", + " 'Sp. Def': 2.2949327864052826e-15,\n", + " 'Speed': 1.049016311882451e-18,\n", + " 'Total': 9.357954335957446e-47}\n", + " ```" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here\n", + "legendary = pokemon[pokemon['Legendary'] == True]\n", + "non_legendary = pokemon[pokemon['Legendary'] == False]" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'HP': 3.330647684846191e-15,\n", + " 'Attack': 7.827253003205333e-24,\n", + " 'Defense': 1.5842226094427255e-12,\n", + " 'Sp. Atk': 6.314915770427266e-41,\n", + " 'Sp. Def': 1.8439809580409594e-26,\n", + " 'Speed': 2.3540754436898437e-21,\n", + " 'Total': 3.0952457469652825e-52}" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "t_test_features(legendary, non_legendary, features=['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed', 'Total'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### From the test results above, what conclusion can you make? Do Legendary and non-Legendary pokemons have significantly different stats on each feature?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your comment here\n", + "#None of the features between the two types of pokemons have the same mean value" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Next, conduct t-test for Generation 1 and Generation 2 pokemons." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'HP': 0.13791881412813622,\n", + " 'Attack': 0.24050968418101457,\n", + " 'Defense': 0.5407630349194362,\n", + " 'Sp. Atk': 0.14119788176331508,\n", + " 'Sp. Def': 0.16781226231606386,\n", + " 'Speed': 0.0028356954812578704,\n", + " 'Total': 0.5599140649014442}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here\n", + "gen_1 = pokemon[pokemon['Generation']== 1]\n", + "gen_2 = pokemon[pokemon['Generation']== 2]\n", + "t_test_features(gen_1, gen_2, features=['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed', 'Total'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What conclusions can you make?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your comment here\n", + "# except 'Speed' feature, the other features between Generation 1 and Generation 2 have no significant difference" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Compare pokemons who have single type vs those having two types." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "# Your code here\n", + "single_type = pokemon[pokemon['Type 2'].isna()]\n", + "double_type = pokemon[pokemon['Type 2'].notna()]" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'HP': 0.11314389855379414,\n", + " 'Attack': 0.00014932578145948305,\n", + " 'Defense': 2.7978540411514693e-08,\n", + " 'Sp. Atk': 0.00013876216585667907,\n", + " 'Sp. Def': 0.00010730610934512779,\n", + " 'Speed': 0.02421703281819093,\n", + " 'Total': 1.1157056505229961e-07}" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "t_test_features(single_type, double_type, features=['HP', 'Attack', 'Defense', 'Sp. Atk', 'Sp. Def', 'Speed', 'Total'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What conclusions can you make?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your comment here\n", + "#Features 'Attack', 'Sp.Atk', 'Sp.Def' and 'Speed' between single-type pokemons and double-type pokemons are significant different; the others 'HP','Defense', and 'Total' are not significantly different" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Now, we want to compare whether there are significant differences of `Attack` vs `Defense` and `Sp. Atk` vs `Sp. Def` of all pokemons. Please write your code below.\n", + "\n", + "*Hint: are you comparing different populations or the same population?*" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "TtestResult(statistic=4.325566393330478, pvalue=1.7140303479358558e-05, df=799)" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "st.ttest_rel(pokemon['Attack'], pokemon['Defense'])" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "TtestResult(statistic=0.853986188453353, pvalue=0.3933685997548122, df=799)" + ] + }, + "execution_count": 27, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "st.ttest_rel(pokemon['Sp. Atk'], pokemon['Sp. Def'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### What conclusions can you make?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your comment here\n", + "# there are no significant differences of `Attack` vs `Defense` and `Sp. Atk` vs `Sp. Def` of all pokemons" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/your-code/challenge-2.ipynb b/your-code/challenge-2.ipynb index 748080d..2e1b5b8 100644 --- a/your-code/challenge-2.ipynb +++ b/your-code/challenge-2.ipynb @@ -1,159 +1,379 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Challenge 2 - ANOVA\n", - "\n", - "In statistics, **Analysis of Variance (ANOVA)** is also used to analyze the differences among group means. The difference between t-test and ANOVA is the former is ued to compare two groups whereas the latter is used to compare three or more groups. [Read more about the difference between t-test and ANOVA](http://b.link/anova24).\n", - "\n", - "From the ANOVA test, you receive two numbers. The first number is called the **F-value** which indicates whether your null-hypothesis can be rejected. The critical F-value that rejects the null-hypothesis varies according to the number of total subjects and the number of subject groups in your experiment. In [this table](http://b.link/eda14) you can find the critical values of the F distribution. **If you are confused by the massive F-distribution table, don't worry. Skip F-value for now and study it at a later time. In this challenge you only need to look at the p-value.**\n", - "\n", - "The p-value is another number yielded by ANOVA which already takes the number of total subjects and the number of experiment groups into consideration. **Typically if your p-value is less than 0.05, you can declare the null-hypothesis is rejected.**\n", - "\n", - "In this challenge, we want to understand whether there are significant differences among various types of pokemons' `Total` value, i.e. Grass vs Poison vs Fire vs Dragon... There are many types of pokemons which makes it a perfect use case for ANOVA." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Import libraries\n", - "import pandas as pd" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Load the data:\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**To achieve our goal, we use three steps:**\n", - "\n", - "1. **Extract the unique values of the pokemon types.**\n", - "\n", - "1. **Select dataframes for each unique pokemon type.**\n", - "\n", - "1. **Conduct ANOVA analysis across the pokemon types.**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### First let's obtain the unique values of the pokemon types. These values should be extracted from Type 1 and Type 2 aggregated. Assign the unique values to a variable called `unique_types`.\n", - "\n", - "*Hint: the correct number of unique types is 19 including `NaN`. You can disregard `NaN` in next step.*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n", - "\n", - "\n", - "len(unique_types) # you should see 19" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Second we will create a list named `pokemon_totals` to contain the `Total` values of each unique type of pokemons.\n", - "\n", - "Why we use a list instead of a dictionary to store the pokemon `Total`? It's because ANOVA only tells us whether there is a significant difference of the group means but does not tell which group(s) are significantly different. Therefore, we don't need know which `Total` belongs to which pokemon type.\n", - "\n", - "*Hints:*\n", - "\n", - "* Loop through `unique_types` and append the selected type's `Total` to `pokemon_groups`.\n", - "* Skip the `NaN` value in `unique_types`. `NaN` is a `float` variable which you can find out by using `type()`. The valid pokemon type values are all of the `str` type.\n", - "* At the end, the length of your `pokemon_totals` should be 18." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "pokemon_totals = []\n", - "\n", - "# Your code here\n", - "\n", - "len(pokemon_totals) # you should see 18" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Now we run ANOVA test on `pokemon_totals`.\n", - "\n", - "*Hints:*\n", - "\n", - "* To conduct ANOVA, you can use `scipy.stats.f_oneway()`. Here's the [reference](http://b.link/scipy44).\n", - "\n", - "* What if `f_oneway` throws an error because it does not accept `pokemon_totals` as a list? The trick is to add a `*` in front of `pokemon_totals`, e.g. `stats.f_oneway(*pokemon_groups)`. This trick breaks the list and supplies each list item as a parameter for `f_oneway`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your code here\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Interpret the ANOVA test result. Is the difference significant?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Your comment here" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 2 - ANOVA\n", + "\n", + "In statistics, **Analysis of Variance (ANOVA)** is also used to analyze the differences among group means. The difference between t-test and ANOVA is the former is ued to compare two groups whereas the latter is used to compare three or more groups. [Read more about the difference between t-test and ANOVA](http://b.link/anova24).\n", + "\n", + "From the ANOVA test, you receive two numbers. The first number is called the **F-value** which indicates whether your null-hypothesis can be rejected. The critical F-value that rejects the null-hypothesis varies according to the number of total subjects and the number of subject groups in your experiment. In [this table](http://b.link/eda14) you can find the critical values of the F distribution. **If you are confused by the massive F-distribution table, don't worry. Skip F-value for now and study it at a later time. In this challenge you only need to look at the p-value.**\n", + "\n", + "The p-value is another number yielded by ANOVA which already takes the number of total subjects and the number of experiment groups into consideration. **Typically if your p-value is less than 0.05, you can declare the null-hypothesis is rejected.**\n", + "\n", + "In this challenge, we want to understand whether there are significant differences among various types of pokemons' `Total` value, i.e. Grass vs Poison vs Fire vs Dragon... There are many types of pokemons which makes it a perfect use case for ANOVA." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Import libraries\n", + "import pandas as pd\n", + "import scipy.stats as st" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
#NameType 1Type 2TotalHPAttackDefenseSp. AtkSp. DefSpeedGenerationLegendary
01BulbasaurGrassPoison3184549496565451False
12IvysaurGrassPoison4056062638080601False
23VenusaurGrassPoison525808283100100801False
33VenusaurMega VenusaurGrassPoison62580100123122120801False
44CharmanderFireNaN3093952436050651False
\n", + "
" + ], + "text/plain": [ + " # Name Type 1 Type 2 Total HP Attack Defense \\\n", + "0 1 Bulbasaur Grass Poison 318 45 49 49 \n", + "1 2 Ivysaur Grass Poison 405 60 62 63 \n", + "2 3 Venusaur Grass Poison 525 80 82 83 \n", + "3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 \n", + "4 4 Charmander Fire NaN 309 39 52 43 \n", + "\n", + " Sp. Atk Sp. Def Speed Generation Legendary \n", + "0 65 65 45 1 False \n", + "1 80 80 60 1 False \n", + "2 100 100 80 1 False \n", + "3 122 120 80 1 False \n", + "4 60 50 65 1 False " + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "RangeIndex: 800 entries, 0 to 799\n", + "Data columns (total 13 columns):\n", + " # Column Non-Null Count Dtype \n", + "--- ------ -------------- ----- \n", + " 0 # 800 non-null int64 \n", + " 1 Name 800 non-null object\n", + " 2 Type 1 800 non-null object\n", + " 3 Type 2 414 non-null object\n", + " 4 Total 800 non-null int64 \n", + " 5 HP 800 non-null int64 \n", + " 6 Attack 800 non-null int64 \n", + " 7 Defense 800 non-null int64 \n", + " 8 Sp. Atk 800 non-null int64 \n", + " 9 Sp. Def 800 non-null int64 \n", + " 10 Speed 800 non-null int64 \n", + " 11 Generation 800 non-null int64 \n", + " 12 Legendary 800 non-null bool \n", + "dtypes: bool(1), int64(9), object(3)\n", + "memory usage: 75.9+ KB\n" + ] + }, + { + "data": { + "text/plain": [ + "None" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Load the data:\n", + "pokemon = pd.read_csv('Pokemon.csv')\n", + "display(pokemon.head())\n", + "display(pokemon.info())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**To achieve our goal, we use three steps:**\n", + "\n", + "1. **Extract the unique values of the pokemon types.**\n", + "\n", + "1. **Select dataframes for each unique pokemon type.**\n", + "\n", + "1. **Conduct ANOVA analysis across the pokemon types.**" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### First let's obtain the unique values of the pokemon types. These values should be extracted from Type 1 and Type 2 aggregated. Assign the unique values to a variable called `unique_types`.\n", + "\n", + "*Hint: the correct number of unique types is 19 including `NaN`. You can disregard `NaN` in next step.*" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "19" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here\n", + "unique_values = pd.concat([pokemon['Type 1'], pokemon['Type 2']]).unique()\n", + "\n", + "len(unique_values)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Second we will create a list named `pokemon_totals` to contain the `Total` values of each unique type of pokemons.\n", + "\n", + "Why we use a list instead of a dictionary to store the pokemon `Total`? It's because ANOVA only tells us whether there is a significant difference of the group means but does not tell which group(s) are significantly different. Therefore, we don't need know which `Total` belongs to which pokemon type.\n", + "\n", + "*Hints:*\n", + "\n", + "* Loop through `unique_types` and append the selected type's `Total` to `pokemon_groups`.\n", + "* Skip the `NaN` value in `unique_types`. `NaN` is a `float` variable which you can find out by using `type()`. The valid pokemon type values are all of the `str` type.\n", + "* At the end, the length of your `pokemon_totals` should be 18." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "18" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "pokemon_totals = []\n", + "\n", + "# Your code here\n", + "for type in unique_values:\n", + " if isinstance(type, str):\n", + " df_type = pokemon[(pokemon['Type 1'] == type)| (pokemon['Type 2'] == type)]\n", + " pokemon_totals.append(df_type['Total'])\n", + " else:\n", + " pass\n", + "len(pokemon_totals) # you should see 18" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Now we run ANOVA test on `pokemon_totals`.\n", + "\n", + "*Hints:*\n", + "\n", + "* To conduct ANOVA, you can use `scipy.stats.f_oneway()`. Here's the [reference](http://b.link/scipy44).\n", + "\n", + "* What if `f_oneway` throws an error because it does not accept `pokemon_totals` as a list? The trick is to add a `*` in front of `pokemon_totals`, e.g. `stats.f_oneway(*pokemon_groups)`. This trick breaks the list and supplies each list item as a parameter for `f_oneway`." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "F_onewayResult(statistic=6.6175382960055344, pvalue=2.6457458815984803e-15)" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here\n", + "st.f_oneway(*pokemon_totals)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Interpret the ANOVA test result. Is the difference significant?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your comment here\n", + "#The feature of 'Total' among the unique types are significantly different as p-value is much lower than 0.05" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}