gabspereira/workspace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e5c2f9889ffa5eabf818464217324986d24f3f43
workspace/data-engineering/01-postgres-modeling/etl.ipynb
@gabriel.pereira 6796398924 refactor: restructure monorepo for clean portfolio layout 
- Move timesfm-forecast into apps/ directory
- Flatten Udacity portfolio projects from deep URL-encoded paths
  into data-engineering/01-XX numbered directories
- Remove old My-Data-Engineering-Portifolio/ parent directory
- Rewrite root README.md: professional overview with badges,
  project table, and repo structure diagram
- Create data-engineering/README.md with per-project descriptions
- Add README.md for 02-cassandra-modeling (was missing)
- Add README.md for 05-airflow-pipelines (was missing)
- Normalize capstone readme.md -> README.md
- Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat,
  Jupyter checkpoints, IDE dirs; remove uv.lock exclusion
- Add dwh.cfg.example and dl.cfg.example credential templates
- Untrack real credential files (dwh.cfg, dl.cfg)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-26 16:48:50 -03:00

1115 lines
39 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"# ETL Processes\n",
"Use this notebook to develop the ETL process for each of your tables before completing the `etl.py` file to load the whole datasets."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"import os\n",
"import glob\n",
"import psycopg2\n",
"import pandas as pd\n",
"from sql_queries import *"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"conn = psycopg2.connect(\"host=127.0.0.1 dbname=sparkifydb user=student password=student\")\n",
"cur = conn.cursor()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"def get_files(filepath):\n",
" all_files = []\n",
" for root, dirs, files in os.walk(filepath):\n",
" files = glob.glob(os.path.join(root,'*.json'))\n",
" for f in files :\n",
" all_files.append(os.path.abspath(f))\n",
" \n",
" return all_files"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"# Process `song_data`\n",
"In this first part, you'll perform ETL on the first dataset, `song_data`, to create the `songs` and `artists` dimensional tables.\n",
"\n",
"Let's perform ETL on a single song file and load a single record into each table to start.\n",
"- Use the `get_files` function provided above to get a list of all song JSON files in `data/song_data`\n",
"- Select the first song in this list\n",
"- Read the song file and view the data"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"song_files = get_files('data/song_data')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"filepath = song_files[0]"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/plain": [
"num_songs 1\n",
"artist_id ARD7TVE1187B99BFB1\n",
"artist_latitude None\n",
"artist_longitude None\n",
"artist_location California - LA\n",
"artist_name Casual\n",
"song_id SOMZWCG12A8C13C480\n",
"title I Didn't Mean To\n",
"duration 218.932\n",
"year 0\n",
"dtype: object"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_json(filepath, typ='series')\n",
"df.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"## #1: `songs` Table\n",
"#### Extract Data for Songs Table\n",
"- Select columns for song ID, title, artist ID, year, and duration\n",
"- Use `df.values` to select just the values from the dataframe\n",
"- Index to select the first (only) record in the dataframe\n",
"- Convert the array to a list and set it to `song_data`"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/plain": [
"array(['SOMZWCG12A8C13C480', \"I Didn't Mean To\", 'ARD7TVE1187B99BFB1', 0,\n",
" 218.93179], dtype=object)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"song_data = df[['song_id','title','artist_id','year','duration']].values\n",
"song_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"#### Insert Record into Song Table\n",
"Implement the `song_table_insert` query in `sql_queries.py` and run the cell below to insert a record for this song into the `songs` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `songs` table in the sparkify database."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"cur.execute(song_table_insert, song_data)\n",
"conn.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"Run `test.ipynb` to see if you've successfully added a record to this table."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"## #2: `artists` Table\n",
"#### Extract Data for Artists Table\n",
"- Select columns for artist ID, name, location, latitude, and longitude\n",
"- Use `df.values` to select just the values from the dataframe\n",
"- Index to select the first (only) record in the dataframe\n",
"- Convert the array to a list and set it to `artist_data`"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/plain": [
"artist_id ARD7TVE1187B99BFB1\n",
"artist_name Casual\n",
"artist_location California - LA\n",
"artist_latitude None\n",
"artist_longitude None\n",
"dtype: object"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"artist_data = df[['artist_id', 'artist_name', 'artist_location', 'artist_latitude', 'artist_longitude']]\n",
"artist_data"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"#### Insert Record into Artist Table\n",
"Implement the `artist_table_insert` query in `sql_queries.py` and run the cell below to insert a record for this song's artist into the `artists` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `artists` table in the sparkify database."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"cur.execute(artist_table_insert, artist_data)\n",
"conn.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"Run `test.ipynb` to see if you've successfully added a record to this table."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"# Process `log_data`\n",
"In this part, you'll perform ETL on the second dataset, `log_data`, to create the `time` and `users` dimensional tables, as well as the `songplays` fact table.\n",
"\n",
"Let's perform ETL on a single log file and load a single record into each table.\n",
"- Use the `get_files` function provided above to get a list of all log JSON files in `data/log_data`\n",
"- Select the first log file in this list\n",
"- Read the log file and view the data"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"log_files = get_files('data/log_data')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"filepath = log_files[0]"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>artist</th>\n",
" <th>auth</th>\n",
" <th>firstName</th>\n",
" <th>gender</th>\n",
" <th>itemInSession</th>\n",
" <th>lastName</th>\n",
" <th>length</th>\n",
" <th>level</th>\n",
" <th>location</th>\n",
" <th>method</th>\n",
" <th>page</th>\n",
" <th>registration</th>\n",
" <th>sessionId</th>\n",
" <th>song</th>\n",
" <th>status</th>\n",
" <th>ts</th>\n",
" <th>userAgent</th>\n",
" <th>userId</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Stephen Lynch</td>\n",
" <td>Logged In</td>\n",
" <td>Jayden</td>\n",
" <td>M</td>\n",
" <td>0</td>\n",
" <td>Bell</td>\n",
" <td>182.85669</td>\n",
" <td>free</td>\n",
" <td>Dallas-Fort Worth-Arlington, TX</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540992e+12</td>\n",
" <td>829</td>\n",
" <td>Jim Henson's Dead</td>\n",
" <td>200</td>\n",
" <td>1543537327796</td>\n",
" <td>Mozilla/5.0 (compatible; MSIE 10.0; Windows NT...</td>\n",
" <td>91</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Manowar</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>0</td>\n",
" <td>Klein</td>\n",
" <td>247.56200</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Shell Shock</td>\n",
" <td>200</td>\n",
" <td>1543540121796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Morcheeba</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>1</td>\n",
" <td>Klein</td>\n",
" <td>257.41016</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Women Lose Weight (Feat: Slick Rick)</td>\n",
" <td>200</td>\n",
" <td>1543540368796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Maroon 5</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>2</td>\n",
" <td>Klein</td>\n",
" <td>231.23546</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Won't Go Home Without You</td>\n",
" <td>200</td>\n",
" <td>1543540625796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Train</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>3</td>\n",
" <td>Klein</td>\n",
" <td>216.76363</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Hey_ Soul Sister</td>\n",
" <td>200</td>\n",
" <td>1543540856796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" artist auth firstName gender itemInSession lastName \\\n",
"0 Stephen Lynch Logged In Jayden M 0 Bell \n",
"1 Manowar Logged In Jacob M 0 Klein \n",
"2 Morcheeba Logged In Jacob M 1 Klein \n",
"3 Maroon 5 Logged In Jacob M 2 Klein \n",
"4 Train Logged In Jacob M 3 Klein \n",
"\n",
" length level location method page \\\n",
"0 182.85669 free Dallas-Fort Worth-Arlington, TX PUT NextSong \n",
"1 247.56200 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"2 257.41016 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"3 231.23546 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"4 216.76363 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"\n",
" registration sessionId song status \\\n",
"0 1.540992e+12 829 Jim Henson's Dead 200 \n",
"1 1.540558e+12 1049 Shell Shock 200 \n",
"2 1.540558e+12 1049 Women Lose Weight (Feat: Slick Rick) 200 \n",
"3 1.540558e+12 1049 Won't Go Home Without You 200 \n",
"4 1.540558e+12 1049 Hey_ Soul Sister 200 \n",
"\n",
" ts userAgent userId \n",
"0 1543537327796 Mozilla/5.0 (compatible; MSIE 10.0; Windows NT... 91 \n",
"1 1543540121796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"2 1543540368796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"3 1543540625796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"4 1543540856796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = pd.read_json(filepath, lines=True)\n",
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"## #3: `time` Table\n",
"#### Extract Data for Time Table\n",
"- Filter records by `NextSong` action\n",
"- Convert the `ts` timestamp column to datetime\n",
" - Hint: the current timestamp is in milliseconds\n",
"- Extract the timestamp, hour, day, week of year, month, year, and weekday from the `ts` column and set `time_data` to a list containing these values in order\n",
" - Hint: use pandas' [`dt` attribute](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.dt.html) to access easily datetimelike properties.\n",
"- Specify labels for these columns and set to `column_labels`\n",
"- Create a dataframe, `time_df,` containing the time data for this file by combining `column_labels` and `time_data` into a dictionary and converting this into a dataframe"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>artist</th>\n",
" <th>auth</th>\n",
" <th>firstName</th>\n",
" <th>gender</th>\n",
" <th>itemInSession</th>\n",
" <th>lastName</th>\n",
" <th>length</th>\n",
" <th>level</th>\n",
" <th>location</th>\n",
" <th>method</th>\n",
" <th>page</th>\n",
" <th>registration</th>\n",
" <th>sessionId</th>\n",
" <th>song</th>\n",
" <th>status</th>\n",
" <th>ts</th>\n",
" <th>userAgent</th>\n",
" <th>userId</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Stephen Lynch</td>\n",
" <td>Logged In</td>\n",
" <td>Jayden</td>\n",
" <td>M</td>\n",
" <td>0</td>\n",
" <td>Bell</td>\n",
" <td>182.85669</td>\n",
" <td>free</td>\n",
" <td>Dallas-Fort Worth-Arlington, TX</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540992e+12</td>\n",
" <td>829</td>\n",
" <td>Jim Henson's Dead</td>\n",
" <td>200</td>\n",
" <td>1543537327796</td>\n",
" <td>Mozilla/5.0 (compatible; MSIE 10.0; Windows NT...</td>\n",
" <td>91</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Manowar</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>0</td>\n",
" <td>Klein</td>\n",
" <td>247.56200</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Shell Shock</td>\n",
" <td>200</td>\n",
" <td>1543540121796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Morcheeba</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>1</td>\n",
" <td>Klein</td>\n",
" <td>257.41016</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Women Lose Weight (Feat: Slick Rick)</td>\n",
" <td>200</td>\n",
" <td>1543540368796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Maroon 5</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>2</td>\n",
" <td>Klein</td>\n",
" <td>231.23546</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Won't Go Home Without You</td>\n",
" <td>200</td>\n",
" <td>1543540625796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Train</td>\n",
" <td>Logged In</td>\n",
" <td>Jacob</td>\n",
" <td>M</td>\n",
" <td>3</td>\n",
" <td>Klein</td>\n",
" <td>216.76363</td>\n",
" <td>paid</td>\n",
" <td>Tampa-St. Petersburg-Clearwater, FL</td>\n",
" <td>PUT</td>\n",
" <td>NextSong</td>\n",
" <td>1.540558e+12</td>\n",
" <td>1049</td>\n",
" <td>Hey_ Soul Sister</td>\n",
" <td>200</td>\n",
" <td>1543540856796</td>\n",
" <td>\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4...</td>\n",
" <td>73</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" artist auth firstName gender itemInSession lastName \\\n",
"0 Stephen Lynch Logged In Jayden M 0 Bell \n",
"1 Manowar Logged In Jacob M 0 Klein \n",
"2 Morcheeba Logged In Jacob M 1 Klein \n",
"3 Maroon 5 Logged In Jacob M 2 Klein \n",
"4 Train Logged In Jacob M 3 Klein \n",
"\n",
" length level location method page \\\n",
"0 182.85669 free Dallas-Fort Worth-Arlington, TX PUT NextSong \n",
"1 247.56200 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"2 257.41016 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"3 231.23546 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"4 216.76363 paid Tampa-St. Petersburg-Clearwater, FL PUT NextSong \n",
"\n",
" registration sessionId song status \\\n",
"0 1.540992e+12 829 Jim Henson's Dead 200 \n",
"1 1.540558e+12 1049 Shell Shock 200 \n",
"2 1.540558e+12 1049 Women Lose Weight (Feat: Slick Rick) 200 \n",
"3 1.540558e+12 1049 Won't Go Home Without You 200 \n",
"4 1.540558e+12 1049 Hey_ Soul Sister 200 \n",
"\n",
" ts userAgent userId \n",
"0 1543537327796 Mozilla/5.0 (compatible; MSIE 10.0; Windows NT... 91 \n",
"1 1543540121796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"2 1543540368796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"3 1543540625796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 \n",
"4 1543540856796 \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4... 73 "
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = df[df['page'].str.contains('NextSong')]\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/plain": [
"0 2018-11-30 00:22:07.796\n",
"1 2018-11-30 01:08:41.796\n",
"2 2018-11-30 01:12:48.796\n",
"3 2018-11-30 01:17:05.796\n",
"4 2018-11-30 01:20:56.796\n",
"Name: ts, dtype: datetime64[ns]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"t = pd.to_datetime(df['ts'], unit='ms')\n",
"t.head()"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"time_data = (t, t.dt.hour, t.dt.day, t.dt.week, t.dt.month, t.dt.year, t.dt.weekday)\n",
"column_labels = ('timestamp', 'hour', 'day', 'week', 'month', 'year', 'weekday')"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"editable": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>timestamp</th>\n",
" <th>hour</th>\n",
" <th>day</th>\n",
" <th>week</th>\n",
" <th>month</th>\n",
" <th>year</th>\n",
" <th>weekday</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2018-11-30 00:22:07.796</td>\n",
" <td>0</td>\n",
" <td>30</td>\n",
" <td>48</td>\n",
" <td>11</td>\n",
" <td>2018</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2018-11-30 01:08:41.796</td>\n",
" <td>1</td>\n",
" <td>30</td>\n",
" <td>48</td>\n",
" <td>11</td>\n",
" <td>2018</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2018-11-30 01:12:48.796</td>\n",
" <td>1</td>\n",
" <td>30</td>\n",
" <td>48</td>\n",
" <td>11</td>\n",
" <td>2018</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2018-11-30 01:17:05.796</td>\n",
" <td>1</td>\n",
" <td>30</td>\n",
" <td>48</td>\n",
" <td>11</td>\n",
" <td>2018</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2018-11-30 01:20:56.796</td>\n",
" <td>1</td>\n",
" <td>30</td>\n",
" <td>48</td>\n",
" <td>11</td>\n",
" <td>2018</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" timestamp hour day week month year weekday\n",
"0 2018-11-30 00:22:07.796 0 30 48 11 2018 4\n",
"1 2018-11-30 01:08:41.796 1 30 48 11 2018 4\n",
"2 2018-11-30 01:12:48.796 1 30 48 11 2018 4\n",
"3 2018-11-30 01:17:05.796 1 30 48 11 2018 4\n",
"4 2018-11-30 01:20:56.796 1 30 48 11 2018 4"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"time_df = pd.DataFrame(dict(zip(column_labels,time_data)))\n",
"time_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"#### Insert Records into Time Table\n",
"Implement the `time_table_insert` query in `sql_queries.py` and run the cell below to insert records for the timestamps in this log file into the `time` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `time` table in the sparkify database."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"for i, row in time_df.iterrows():\n",
" cur.execute(time_table_insert, list(row))\n",
" conn.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"Run `test.ipynb` to see if you've successfully added records to this table."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"## #4: `users` Table\n",
"#### Extract Data for Users Table\n",
"- Select columns for user ID, first name, last name, gender and level and set to `user_df`"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"user_df = df[['userId', 'firstName', 'lastName', 'gender', 'level']]"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"#### Insert Records into Users Table\n",
"Implement the `user_table_insert` query in `sql_queries.py` and run the cell below to insert records for the users in this log file into the `users` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `users` table in the sparkify database."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"editable": true
},
"outputs": [
{
"ename": "IndexError",
"evalue": "index out of bounds",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m 3117\u001b[0m return self._engine.get_value(s, k,\n\u001b[0;32m-> 3118\u001b[0;31m tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m 3119\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mKeyError\u001b[0m: 5",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-20-2c0cc95bc01c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrow\u001b[0m \u001b[0;32min\u001b[0m \u001b[0muser_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miterrows\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcur\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muser_table_insert\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcommit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m 765\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 766\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 767\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 768\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 769\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m 3122\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3123\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3124\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mlibindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 3125\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3126\u001b[0m \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.get_value_box\u001b[0;34m()\u001b[0m\n",
"\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.get_value_box\u001b[0;34m()\u001b[0m\n",
"\u001b[0;31mIndexError\u001b[0m: index out of bounds"
]
}
],
"source": [
"for i, row in user_df.iterrows():\n",
" cur.execute(user_table_insert, row)\n",
" conn.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"Run `test.ipynb` to see if you've successfully added records to this table."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"## #5: `songplays` Table\n",
"#### Extract Data and Songplays Table\n",
"This one is a little more complicated since information from the songs table, artists table, and original log file are all needed for the `songplays` table. Since the log file does not specify an ID for either the song or the artist, you'll need to get the song ID and artist ID by querying the songs and artists tables to find matches based on song title, artist name, and song duration time.\n",
"- Implement the `song_select` query in `sql_queries.py` to find the song ID and artist ID based on the title, artist name, and duration of a song.\n",
"- Select the timestamp, user ID, level, song ID, artist ID, session ID, location, and user agent and set to `songplay_data`\n",
"\n",
"#### Insert Records into Songplays Table\n",
"- Implement the `songplay_table_insert` query and run the cell below to insert records for the songplay actions in this log file into the `songplays` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `songplays` table in the sparkify database."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"df.head(1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"for index, row in df.iterrows():\n",
"\n",
" # get songid and artistid from song and artist tables\n",
" cur.execute(song_select, (row.song, row.artist, row.length))\n",
" results = cur.fetchone()\n",
" \n",
" if results:\n",
" songid, artistid = results\n",
" else:\n",
" songid, artistid = None, None\n",
"\n",
" # insert songplay record\n",
" start_time = pd.to_datetime(row.ts, unit='ms').strftime('%Y-%m-%d %I:%M:%S')\n",
" songplay_data = (index, start_time, row.userId, row.level, str(songid), str(artistid), row.sessionId, row.location, row.userAgent)\n",
" cur.execute(songplay_table_insert, songplay_data)\n",
" conn.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"Run `test.ipynb` to see if you've successfully added records to this table."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"# Close Connection to Sparkify Database"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true
},
"outputs": [],
"source": [
"conn.close()"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true
},
"source": [
"# Implement `etl.py`\n",
"Use what you've completed in this notebook to implement `etl.py`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"editable": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Reference in New Issue View Git Blame Copy Permalink