First commit

Cloud+Data+Warehouse/Project+Data+Warehouse/README.md

@@ -0,0 +1,86 @@
+# Project 3: Song Play Analysis with S3 and Redshift
+-------------------------
+
+### Introduction
+
+In this project, we will help one music streaming startup - Sparkify, to move their user and song database processes to the cloud. To reach that, we build an ETL pipeline to extracts their data from **AWS S3** (data storage), stages tables on **AWS Redshift** (data warehouse with *columnar storage*), and execute **SQL** statements to create the analytics tables from these staging tables.
+
+### Datasets
+Datasets used in this project are provided in two public **S3 buckets**. 
+
++ **Song Dataset** - The first dataset is a subset of real data from the Million Song Dataset. Each file is in JSON format and contains metadata about a song and the artist of that song. The files are partitioned by the first three letters of each song's track ID. For example, here are file paths to two files in this dataset.
+
+```
+song_data/A/B/C/TRABCEI128F424C983.json
+song_data/A/A/B/TRAABJL12903CDCF1A.json
+````
+
+And below is an example of what a single song file, TRAABJL12903CDCF1A.json, looks like.
+
+```
+{"num_songs": 1, "artist_id": "ARJIE2Y1187B994AB7", "artist_latitude": null, "artist_longitude": null, "artist_location": "", "artist_name": "Line Renaud", "song_id": "SOUPIRU12A6D4FA1E1", "title": "Der Kleine Dompfaff", "duration": 152.92036, "year": 0}
+
+```
+
++ **Log Dataset** - The second dataset consists of log files in JSON format generated by this event simulator based on the songs in the dataset above. These simulate app activity logs from an imaginary music streaming app based on configuration settings.
+
+The log files in the dataset you'll be working with are partitioned by year and month. For example, here are file paths to two files in this dataset.
+
+```
+log_data/2018/11/2018-11-12-events.json
+log_data/2018/11/2018-11-13-events.json
+```
+
+And below is an example of what the data in a log file, 2018-11-12-events.json, looks like.
+
+![log-data](./log-data.PNG)
+
+The **Redshift** service is where data will be ingested and transformed, using `COPY` command we will access to the JSON files inside the buckets and copy their content to our *staging tables*.
+
+### Database Schema
+We have two staging tables which *copy* the JSON file inside the **S3 buckets**.
+#### Staging Tables
++ **staging_songs** - info about songs and artists
++ **staging_events** - actions done by users (which song are listening, etc.. )
+
+
+A star schema was designed to optimize queries on song play analysis. This includes the following tables.
+
+#### Fact Table 
++ **songplays** - records in event data associated with song plays i.e. records with page `NextSong`
+
+#### Dimension Tables
++ **users** - users in the app
++ **songs** - songs in music database
++ **artists** - artists in music database
++ **time** - timestamps of records in **songplays** broken down into specific units
+
+
+The database schema is shown as follows
+
+![schema](./schema_diagram.PNG)
+
+### Data Warehouse Configurations and Setup steps:
+* Create a new `IAM user` in your AWS account
+* Give it AdministratorAccess and Attach policies
+* Use access key and secret key to create clients for `EC2`, `S3`, `IAM`, and `Redshift`.
+* Create an `IAM Role` that makes `Redshift` able to access `S3 bucket` (ReadOnly)
+* Create a `RedShift Cluster` and get the `DWH_ENDPOIN(Host address)` and `DWH_ROLE_ARN` and fill the config file.
+
+### ETL Pipeline
++ Created tables to store the data from `S3 buckets`.
++ Loading the data from `S3 buckets` to staging tables in the `Redshift Cluster`.
++ Inserted data into fact and dimension tables from the staging tables.
+
+### Project Structure
+
++ `create_tables.py` - This script will drop old tables (if exist) ad re-create new tables.
++ `etl.py` - This script executes the queries that extract `JSON` data from the `S3 bucket` and ingest them to `Redshift`.
++ `sql_queries.py` - This file contains variables with SQL statement in String formats, partitioned by `CREATE`, `DROP`, `COPY` and `INSERT` statement.
++ `dhw.cfg` - Configuration file used that contains info about `Redshift`, `IAM` and `S3`
+
+### How to Run
+
+1. Create tables by running `create_tables.py`.
+
+2. Execute ETL process by running `etl.py`.

Cloud+Data+Warehouse/Project+Data+Warehouse/create_tables.py

@@ -0,0 +1,40 @@
+import configparser
+import psycopg2
+from sql_queries import create_table_queries, drop_table_queries
+
+
+def drop_tables(cur, conn):
+    for query in drop_table_queries:
+        try:
+            cur.execute(query)
+            conn.commit()
+            print(query + ' - Done!')
+        except:
+            print(query + ' - Fail!')
+            pass
+
+def create_tables(cur, conn):
+    for query in create_table_queries:
+        try:
+            cur.execute(query)
+            conn.commit()
+            print(query + ' - Done!')
+        except:
+            print(query + ' - Fail!')
+            pass
+
+def main():
+    config = configparser.ConfigParser()
+    config.read('dwh.cfg')
+
+    conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
+    cur = conn.cursor()
+
+    drop_tables(cur, conn)
+    create_tables(cur, conn)
+
+    conn.close()
+
+
+if __name__ == "__main__":
+    main()

Cloud+Data+Warehouse/Project+Data+Warehouse/dwh.cfg

@@ -0,0 +1,29 @@
+[AWS]
+KEY=AKIAWVLQ3XEKGAJJ6WNT
+SECRET=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
+
+[CLUSTER]
+HOST=redshift-cluster-1.cfw5ezaysudh.us-west-2.redshift.amazonaws.com
+DB_NAME=dev
+DB_USER=awsuser
+DB_PASSWORD=Yz3bbo4U
+DB_PORT=5439
+
+[IAM_ROLE]
+ARN=arn:aws:iam::995187936370:role/myRedshiftRole
+
+[S3]
+LOG_DATA='s3://udacity-dend/log_data'
+LOG_JSONPATH='s3://udacity-dend/log_json_path.json'
+SONG_DATA='s3://udacity-dend/song_data'
+
+[DWH] 
+DWH_CLUSTER_TYPE=multi-node
+DWH_NUM_NODES=1
+DWH_NODE_TYPE=dc2.large
+DWH_CLUSTER_IDENTIFIER=redshift-cluster-1
+DWH_DB=dev
+DWH_DB_USER=awsuser
+DWH_DB_PASSWORD=Yz3bbo4U
+DWH_PORT=5439
+DWH_IAM_ROLE_NAME=myRedshiftRole

Cloud+Data+Warehouse/Project+Data+Warehouse/etl.py

@@ -0,0 +1,45 @@
+import configparser
+import psycopg2
+from sql_queries import copy_table_queries, insert_table_queries
+
+
+def load_staging_tables(cur, conn):
+    '''Copy the staging tables to S3 bucket: staging_events and staging_songs'''
+    for query in copy_table_queries:
+        try:
+            cur.execute(query)
+            conn.commit()
+            print(query + ' - Done!')
+        except:
+            print(query + ' - Fail!')
+            pass
+
+def insert_tables(cur, conn):
+    '''Load the values from staging table into the tables: songplays, users, songs, artists, time'''
+    for query in insert_table_queries:
+        try:
+            cur.execute(query)
+            conn.commit()
+            print(query + ' - Done!')
+        except:
+            print(query + ' - Fail!')
+            pass
+
+def main():
+    '''Read AWS requirements from dwh.cfg file'''
+    config = configparser.ConfigParser()
+    config.read('dwh.cfg')
+
+    '''Connect to the Redshift cluster'''
+    conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
+    cur = conn.cursor()
+    
+    '''Run above functions according to sql_queries.py'''
+    load_staging_tables(cur, conn)
+    insert_tables(cur, conn)
+
+    conn.close()
+
+
+if __name__ == "__main__":
+    main()

Cloud+Data+Warehouse/Project+Data+Warehouse/sql_queries.py

@@ -0,0 +1,211 @@
+import configparser
+
+
+# CONFIG
+config = configparser.ConfigParser()
+config.read('dwh.cfg')
+
+LOG_DATA  = config.get("S3", "LOG_DATA")
+LOG_PATH  = config.get("S3", "LOG_JSONPATH")
+SONG_DATA = config.get("S3", "SONG_DATA")
+IAM_ROLE  = config.get("IAM_ROLE","ARN")
+
+# DROP TABLES
+
+staging_events_table_drop = "DROP TABLE IF EXISTS staging_events"
+staging_songs_table_drop = "DROP TABLE IF EXISTS staging_songs"
+songplay_table_drop = "DROP TABLE IF EXISTS songplays"
+user_table_drop = "DROP TABLE IF EXISTS users"
+song_table_drop = "DROP TABLE IF EXISTS songs"
+artist_table_drop = "DROP TABLE IF EXISTS artists"
+time_table_drop = "DROP TABLE IF EXISTS time"
+
+# CREATE TABLES
+
+staging_events_table_create= """
+CREATE TABLE IF NOT EXISTS staging_events(
+        artist              text,
+        auth                text,
+        first_name          text,
+        gender              text,
+        ItemInSession       int,
+        last_name           text,
+        length              float,
+        level               text,
+        location            text,
+        method              text,
+        page                text,
+        registration        text,
+        session_id          int,
+        song                text,
+        status              int,
+        ts                  bigint, 
+        user_agent          text, 
+        user_id             int
+)
+"""
+
+staging_songs_table_create = """
+CREATE TABLE IF NOT EXISTS staging_songs(
+        song_id             text,
+        artist_id           text,
+        artist_latitude     float,
+        artist_longitude    float,
+        artist_location     text,
+        artist_name         varchar(255),
+        duration            float,
+        num_songs           int,
+        title               text,
+        year                int
+    )
+"""
+
+songplay_table_create = """
+CREATE TABLE IF NOT EXISTS songplays(
+        songplay_id         int identity(0,1) primary key,
+        start_time          timestamp not null sortkey distkey,
+        user_id             int not null,
+        level               varchar,
+        song_id             varchar,
+        artist_id           varchar,
+        session_id          int,
+        location            varchar,
+        user_agent          varchar
+    )
+"""
+
+user_table_create = """
+CREATE TABLE IF NOT EXISTS users(
+        user_id             varchar PRIMARY KEY,
+        first_name          varchar,
+        last_name           varchar,
+        gender              varchar,
+        level               varchar
+    )
+"""
+
+song_table_create = """
+CREATE TABLE IF NOT EXISTS songs(
+        song_id             varchar PRIMARY KEY NOT NULL,
+        title               varchar NOT NULL,
+        artist_id           varchar NOT NULL,
+        year                int,
+        duration            float
+    )
+"""
+
+artist_table_create = ("""
+CREATE TABLE IF NOT EXISTS artists(
+        artist_id           varchar PRIMARY KEY NOT NULL,
+        name                varchar,
+        location            varchar,
+        latitude            float,
+        longitude           float
+    )
+""")
+
+time_table_create = ("""
+CREATE TABLE IF NOT EXISTS time
+    (
+        start_time          timestamp not null distkey sortkey primary key,
+        hour                int not null,
+        day                 int not null,
+        week                int not null,
+        month               int not null,
+        year                int not null,
+        weekday             varchar not null
+    ) 
+""")
+
+# STAGING TABLES
+
+staging_events_copy = ("""
+copy staging_events from {bucket}
+    credentials 'aws_iam_role={role}'
+    region      'us-west-2'
+    format       as JSON {path}
+    timeformat   as 'epochmillisecs'
+""").format(bucket=LOG_DATA, role=IAM_ROLE, path=LOG_PATH)
+
+staging_songs_copy = ("""
+copy staging_songs from {bucket}
+    credentials 'aws_iam_role={role}'
+    region      'us-west-2'
+    format       as JSON 'auto'
+""").format(bucket=SONG_DATA, role=IAM_ROLE)
+
+# FINAL TABLES
+
+songplay_table_insert = ("""
+    INSERT INTO songplays (start_time, user_id, level, song_id, artist_id, session_id, location, user_agent) SELECT
+        TIMESTAMP 'epoch' + (e.ts/1000 * interval '1 second'),
+        e.user_id,
+        e.level,
+        s.song_id,
+        s.artist_id,
+        e.session_id, 
+        e.location,
+        e.user_agent
+    FROM staging_events e
+    LEFT JOIN staging_songs s ON
+        e.song = s.title AND
+        e.artist = s.artist_name AND
+        e.length = s.duration
+    WHERE
+        e.page = 'NextSong'
+        and e.user_id IS NOT NULL
+""")
+
+user_table_insert = ("""
+    INSERT INTO users SELECT DISTINCT (user_id)
+        user_id,
+        first_name,
+        last_name,
+        gender,
+        level
+    FROM staging_events
+    where user_id IS NOT NULL
+""")
+
+song_table_insert = ("""
+    INSERT INTO songs SELECT DISTINCT (song_id)
+        song_id,
+        title,
+        artist_id,
+        year,
+        duration
+    FROM staging_songs
+""")
+
+artist_table_insert = ("""
+    INSERT INTO artists SELECT DISTINCT (artist_id)
+        artist_id,
+        artist_name,
+        artist_location,
+        artist_latitude,
+        artist_longitude
+    FROM staging_songs
+""")
+
+
+time_table_insert = ("""
+    INSERT INTO time
+        WITH temp_time AS (SELECT TIMESTAMP 'epoch' + (ts/1000 * interval '1 second') as ts FROM staging_events)
+        SELECT DISTINCT
+        ts,
+        extract(hour from ts),
+        extract(day from ts),
+        extract(week from ts),
+        extract(month from ts),
+        extract(year from ts),
+        extract(weekday from ts)
+        FROM temp_time
+""")
+
+
+# QUERY LISTS
+
+create_table_queries = [staging_events_table_create, staging_songs_table_create, songplay_table_create, user_table_create, song_table_create, artist_table_create, time_table_create]
+drop_table_queries = [staging_events_table_drop, staging_songs_table_drop, songplay_table_drop, user_table_drop, song_table_drop, artist_table_drop, time_table_drop]
+copy_table_queries = [staging_events_copy, staging_songs_copy]
+insert_table_queries = [songplay_table_insert, user_table_insert, song_table_insert, artist_table_insert, time_table_insert]