Skip to content

[lab-dataframe-calculations] Berna Ozer #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
berna-ozer wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[lab-dataframe-calculations] Berna Ozer #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
View file
Open in desktop
Binary file not shown.
Binary file added your-code/.DS_Store
View file
Open in desktop
Binary file not shown.
1,675 changes: 1,675 additions & 0 deletions your-code/.ipynb_checkpoints/challenge-1-checkpoint.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

352 changes: 352 additions & 0 deletions your-code/.ipynb_checkpoints/challenge-2-checkpoint.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,352 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Challenge 2\n",
"\n",
"In this challenge we will continue working with the `Pokemon` dataset. We will attempt solving a slightly more complex problem in which we will practice the iterative data analysis process you leaned in [this video](https://www.youtube.com/watch?v=xOomNicqbkk).\n",
"\n",
"The problem statement is as follows:\n",
"\n",
"**You are at a Pokemon black market planning to buy a Pokemon for battle. All Pokemon are sold at the same price and you can only afford to buy one. You cannot choose which specific Pokemon to buy. However, you can specify the type of the Pokemon - one type that exists in either `Type 1` or `Type 2`. Which type should you choose in order to maximize your chance of receiving a good Pokemon?**\n",
"\n",
"To remind you about the 3 steps of iterative data analysis, they are:\n",
"\n",
"1. Setting Expectations\n",
"1. Collecting Information\n",
"1. Reacting to Data / Revising Expectations\n",
"\n",
"Following the iterative process, we'll guide you in completing the challenge."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Problem Solving Iteration 1\n",
"\n",
"In this iteration we'll analyze the problem and identify the breakthrough. The original question statement is kind of vague because we don't know what a *good pokemon* really means as represented in the data. We'll start by understanding the dataset and see if we can find some insights."
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"# Import libraries\n",
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>#</th>\n",
" <th>Name</th>\n",
" <th>Type 1</th>\n",
" <th>Type 2</th>\n",
" <th>Total</th>\n",
" <th>HP</th>\n",
" <th>Attack</th>\n",
" <th>Defense</th>\n",
" <th>Sp. Atk</th>\n",
" <th>Sp. Def</th>\n",
" <th>Speed</th>\n",
" <th>Generation</th>\n",
" <th>Legendary</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>Bulbasaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>318</td>\n",
" <td>45</td>\n",
" <td>49</td>\n",
" <td>49</td>\n",
" <td>65</td>\n",
" <td>65</td>\n",
" <td>45</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>Ivysaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>405</td>\n",
" <td>60</td>\n",
" <td>62</td>\n",
" <td>63</td>\n",
" <td>80</td>\n",
" <td>80</td>\n",
" <td>60</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>Venusaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>525</td>\n",
" <td>80</td>\n",
" <td>82</td>\n",
" <td>83</td>\n",
" <td>100</td>\n",
" <td>100</td>\n",
" <td>80</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3</td>\n",
" <td>VenusaurMega Venusaur</td>\n",
" <td>Grass</td>\n",
" <td>Poison</td>\n",
" <td>625</td>\n",
" <td>80</td>\n",
" <td>100</td>\n",
" <td>123</td>\n",
" <td>122</td>\n",
" <td>120</td>\n",
" <td>80</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>4</td>\n",
" <td>Charmander</td>\n",
" <td>Fire</td>\n",
" <td>NaN</td>\n",
" <td>309</td>\n",
" <td>39</td>\n",
" <td>52</td>\n",
" <td>43</td>\n",
" <td>60</td>\n",
" <td>50</td>\n",
" <td>65</td>\n",
" <td>1</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" # Name Type 1 Type 2 Total HP Attack Defense \\\n",
"0 1 Bulbasaur Grass Poison 318 45 49 49 \n",
"1 2 Ivysaur Grass Poison 405 60 62 63 \n",
"2 3 Venusaur Grass Poison 525 80 82 83 \n",
"3 3 VenusaurMega Venusaur Grass Poison 625 80 100 123 \n",
"4 4 Charmander Fire NaN 309 39 52 43 \n",
"\n",
" Sp. Atk Sp. Def Speed Generation Legendary \n",
"0 65 65 45 1 False \n",
"1 80 80 60 1 False \n",
"2 100 100 80 1 False \n",
"3 122 120 80 1 False \n",
"4 60 50 65 1 False "
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Importing the dataset\n",
"pokemon = pd.read_csv('Pokemon.csv')\n",
"pokemon.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the data it seems whether a pokemon is good depends on its abilities as represented in the fields of `HP`, `Attack`, `Defense`, `Sp. Atk`, `Sp. Def`, `Speed`, and `Total`. We are not sure about `Generation` and `Legendary` because they are not necessarily the decisive factors of the pokemon abilities.\n",
"\n",
"But `HP`, `Attack`, `Defense`, `Sp. Atk`, `Sp. Def`, `Speed`, and `Total` are a lot of fields! If we look at them all at once it's very complicated. This isn't Mission Impossible but it's ideal that we tackle this kind of problem after we learn Machine Learning (which you will do in Module 3). For now, is there a way to consolidate the fields we need to look into?\n",
"\n",
"Fortunately there seems to be a way. It appears the `Total` field is computed based on the other 6 fields. But we need to prove our theory. If we can approve there is a formula to compute `Total` based on the other 6 abilities, we only need to look into `Total`.\n",
"\n",
"We have the following expectation now:\n",
"\n",
"#### The `Total` field is computed based on `HP`, `Attack`, `Defense`, `Sp. Atk`, `Sp. Def`, and `Speed`.\n",
"\n",
"We need to collect the following information:\n",
"\n",
"* **What is the formula to compute `Total`?**\n",
"* **Does the formula work for all pokemon?**\n",
"\n",
"In the cell below, make a hypothesis on how `Total` is computed and test your hypothesis."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# your code here\n",
"pokemon['Total_hypo'] = pokemon['HP'] + pokemon['Attack'] + pokemon['Defense'] + pokemon['Sp. Atk'] + pokemon['Sp. Def'] + pokemon['Speed']\n",
"(pokemon['Total'] == pokemon['Total_hypo']).sum()==len(pokemon['Total'])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Problem Solving Iteration 2\n",
"\n",
"Now that we have consolidated the abilities fields, we can update the problem statement. The new problem statement is:\n",
"\n",
"### Which pokemon type is most likely to have the highest `Total` value?\n",
"\n",
"In the updated problem statement, we assume there is a certain relationship between the `Total` and the pokemon type. But we have two *type* fields (`Type 1` and `Type 2`) that have string values. In data analysis, string fields have to be transformed to numerical format in order to be analyzed. \n",
"\n",
"In addition, keep in mind that `Type 1` always has a value but `Type 2` is sometimes empty (having the `NaN` value). Also, the pokemon type we choose may be either in `Type 1` or `Type 2`.\n",
"\n",
"Now our expectation is:\n",
"\n",
"#### `Type 1` and `Type 2` string variables need to be converted to numerical variables in order to identify the relationship between `Total` and the pokemon type.\n",
"\n",
"The information we need to collect is:\n",
"\n",
"#### How to convert two string variables to numerical?\n",
"\n",
"Let's address the first question first. You can use a method called **One Hot Encoding** which is frequently used in machine learning to encode categorical string variables to numerical. The idea is to gather all the possible string values in a categorical field and create a numerical field for each unique string value. Each of those numerical fields uses `1` and `0` to indicate whether the data record has the corresponding categorical value. A detailed explanation of One Hot Encoding can be found in [this article](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f). You will formally learn it in Module 3.\n",
"\n",
"For instance, if a pokemon has `Type 1` as `Poison` and `Type 2` as `Fire`, then its `Poison` and `Fire` fields are `1` whereas all other fields are `0`. If a pokemon has `Type 1` as `Water` and `Type 2` as `NaN`, then its `Water` field is `1` whereas all other fields are `0`.\n",
"\n",
"#### In the next cell, use One Hot Encoding to encode `Type 1` and `Type 2`. Use the pokemon type values as the names of the numerical fields you create.\n",
"\n",
"The new numerical variables you create should look like below:\n",
"\n",
"![One Hot Encoding](../images/one-hot-encoding.png)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Problem Solving Iteration 3\n",
"\n",
"Now we have encoded the pokemon types, we will identify the relationship between `Total` and the encoded fields. Our expectation is:\n",
"\n",
"#### There are relationships between `Total` and the encoded pokemon type variables and we need to identify the correlations.\n",
"\n",
"The information we need to collect is:\n",
"\n",
"#### How to identify the relationship between `Total` and the encoded pokemon type fields?\n",
"\n",
"There are multiple ways to answer this question. The easiest way is to use correlation. In the cell below, calculate the correlation of `Total` to each of the encoded fields. Rank the correlations and identify the #1 pokemon type that is most likely to have the highest `Total`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# your code here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Bonus Question\n",
"\n",
"Say now you can choose both `Type 1` and `Type 2` of the pokemon. In order to receive the best pokemon, which types will you choose?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# your code here"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading