Skip to content

Updated webscraping #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
isi-mube wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Updated webscraping #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
281 changes: 281 additions & 0 deletions s-307/API.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,281 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<center>\n",
" <img src=\"files/logo2.png\" width=\"200\" />\n",
"\n",
"<h1 style=\"color: #00BFFF;\">\"Data Hunting and Gathering\"</h1>\n",
"\n",
"<img src=\"files/cavalls-de-valltorta.jpg\" width=\"400\" />\n",
"</center>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h2 style=\"color: #00BFFF;\">Introduction to API</h2>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## APIS (Aplication Programing Interface)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Definitions\n",
"\n",
"#### URLs\n",
"\n",
"**U**niform **R**esource **L**ocator\n",
"\n",
"Contains the information about a resource we (the CLIENT) are requesting from a SERVER\n",
"\n",
"http://www.google.com/search?q=puppies\n",
"\n",
"http://127.0.0.1:306/invocations\n",
"\n",
"- Protocol: http\n",
"- Top Level Domain: com\n",
"- Domain: google\n",
"- Subdomain: www\n",
"- IP: 127.0.0.1\n",
"- Port: 306\n",
"- Route/Folder/Path: search/invocations\n",
"- Query Parameters: q=puppies\n",
"\n",
"#### HTTP\n",
"**H**yper **T**ext **T**transfer **P**rotocol (**S**ecure) \n",
"\n",
"HTTP(S) is a protocol that provides a structure for request between a client and a server.\n",
"For example, the web browser of a user (the client) uses HTTP to request information from a server that hoist a website\n",
"\n",
"#### Response\n",
"The response is usually dependent on the functionality you are looking for:\n",
" * a JSON \n",
" * an image\n",
" * a video\n",
" * an HTML\n",
" * ...\n",
"\n",
"#### Request\n",
"**Requests** are the questions we (clients) ask of a server to get some information (the **response**). \n",
"Types of request (verbs):\n",
" * GET: read info from a resource and do not change it in any way. This is the standard request that in most sites gets the HTML+CSS of the page as a response.\n",
" * POST: send data that creates/updates a resource, or triggers some process.\n",
" * PUT\n",
" * DELETE\n",
" * PATCH\n",
" * ...\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"TFL = requests.get('https://api.tfl.gov.uk/AirQuality')\n",
"\n",
"TFL.headers\n",
"TFL.content\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## Content is typically in a JSON format - What does a json look like?\n",
"TFL.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![json](https://www.convertsimple.com/wp-content/uploads/2022/05/json-example.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Onboarding API data into Pandas\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"weather_data = pd.DataFrame.from_dict(TFL.json())\n",
"weather_data.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Not ideal, part of the request is still in json. This is a nested json...\n",
"weather_data['currentForecast'][1]\n",
"\n",
"# There is a function in pandas to un-nest jsons, but it makes some assumptions and sometimes we have to unpack hierarchical structures ourselves\n",
"# beware this usually involves a lot of for loops and apply functions\n",
"pd.json_normalize(weather_data['currentForecast'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Parameters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"r = requests.get('https://v2.jokeapi.dev/joke/programming')\n",
"r.json()\n",
"\n",
"\n",
"# Sometimes we want to pass parameters to the endpoint, just like we pass arguments to functions in python \n",
"# We pass parameters via the url as `?param1=value1&param2=value2...` at the end of the url\n",
"r = requests.get('https://v2.jokeapi.dev/joke/programming?contains=python&amount=3')\n",
"r.json()\n",
"\n",
"params_dict = {\"contains\":\"python\",\"amount\":\"3\"}\n",
"r = requests.get('https://v2.jokeapi.dev/joke/programming',params=params_dict)\n",
"r.json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Business Challange API"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example: Fetching Weather Data Using OpenWeatherMap API in Python\n",
"\n",
"This example demonstrates how to use the OpenWeatherMap API to fetch current weather data for a specific city using Python.\n",
"\n",
"#### Prerequisites\n",
"- An API key from OpenWeatherMap.\n",
"- Python's `requests` library installed. (Install via `pip install requests` if needed.)\n",
"\n",
"#### Steps to Follow\n",
"1. **Sign Up for OpenWeatherMap API**:\n",
" - Register for an account at [OpenWeatherMap](https://openweathermap.org/api).\n",
" - Obtain your free API key (note that there might be an activation delay).\n",
"\n",
"2. **Python Script for Weather Data Retrieval**:\n",
" - The script uses the `requests` library to make an API call.\n",
" - Replace `'YOUR_API_KEY'` with your actual OpenWeatherMap API key.\n",
" - Replace `'CITY_NAME'` with your desired city name."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Business Challenge II - Optional\n",
"\n",
"#### Challenge: Analyzing Instagram Hashtag Trends with Instaloader\n",
"\n",
"## Objective\n",
"Leverage `Instaloader`, a Python library, to download posts associated with a specific hashtag on Instagram. Analyze the collected data to identify trends, popular content, and user engagement.\n",
"\n",
"## Steps\n",
"\n",
"### 1. Install Instaloader\n",
"- Ensure Python is installed on your system.\n",
"- Install `Instaloader` using pip: `pip install instaloader`\n",
"\n",
"\n",
"### 2. Data Collection\n",
"- Choose a hashtag relevant to a topic of interest (e.g., #nature, #travel, #food).\n",
"- Use `Instaloader` to download posts tagged with the chosen hashtag. Consider limitations like the number of posts to avoid overwhelming the API.\n",
"\n",
"```python\n",
"import instaloader\n",
"\n",
"L = instaloader.Instaloader()\n",
"posts = instaloader.Hashtag.from_name(L.context, 'YOUR_HASHTAG').get_posts()\n",
"\n",
"for post in posts:\n",
" # Add code to process and store post details\n",
"```\n",
"### 3. Data Analysis\n",
"Analyze the downloaded data for:\n",
"- Popular trends in the hashtag.\n",
"- Common themes or subjects in images or captions.\n",
"- Levels of user engagement (likes, comments).\n",
"\n",
"### 4. Reporting\n",
"- Compile your findings into a report.\n",
"- Include visual representations (graphs, word clouds) to illustrate key trends.\n",
"\n",
"### Important Notes\n",
"- Respect Instagram's terms of service and ethical guidelines in data scraping.\n",
"- Be mindful of privacy and consent, especially with user-generated content.\n",
"- The scope of data collection should be limited for educational purposes.\n",
"\n",
"### Expected Outcome\n",
"This challenge aims to provide practical experience with Instaloader, develop data analysis skills, and offer insights into social media trends and user behavior."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"\n",
"def get_weather(api_key, city):\n",
" base_url = \"http://api.openweathermap.org/data/2.5/weather?\"\n",
" city_name = city\n",
" complete_url = f\"{base_url}appid={api_key}&q={city_name}\"\n",
" response = requests.get(complete_url)\n",
" return response.json()\n",
"\n",
"# Replace 'YOUR_API_KEY' with your actual API key and 'CITY_NAME' with your city\n",
"api_key = 'YOUR_API_KEY'\n",
"city_name = 'CITY_NAME'\n",
"weather_data = get_weather(api_key, city_name)\n",
"\n",
"print(f\"Weather in {city_name}:\")\n",
"print(weather_data)"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading