refactor: restructure monorepo for clean portfolio layout

- Move timesfm-forecast into apps/ directory
- Flatten Udacity portfolio projects from deep URL-encoded paths
  into data-engineering/01-XX numbered directories
- Remove old My-Data-Engineering-Portifolio/ parent directory
- Rewrite root README.md: professional overview with badges,
  project table, and repo structure diagram
- Create data-engineering/README.md with per-project descriptions
- Add README.md for 02-cassandra-modeling (was missing)
- Add README.md for 05-airflow-pipelines (was missing)
- Normalize capstone readme.md -> README.md
- Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat,
  Jupyter checkpoints, IDE dirs; remove uv.lock exclusion
- Add dwh.cfg.example and dl.cfg.example credential templates
- Untrack real credential files (dwh.cfg, dl.cfg)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@gabriel.pereira
2026-03-26 16:48:50 -03:00
parent 5c4e6075e1
commit 6796398924
160 changed files with 308 additions and 34 deletions

View File

@@ -0,0 +1,45 @@
import configparser
import psycopg2
from sql_queries import copy_table_queries, insert_table_queries
def load_staging_tables(cur, conn):
'''Copy the staging tables to S3 bucket: staging_events and staging_songs'''
for query in copy_table_queries:
try:
cur.execute(query)
conn.commit()
print(query + ' - Done!')
except:
print(query + ' - Fail!')
pass
def insert_tables(cur, conn):
'''Load the values from staging table into the tables: songplays, users, songs, artists, time'''
for query in insert_table_queries:
try:
cur.execute(query)
conn.commit()
print(query + ' - Done!')
except:
print(query + ' - Fail!')
pass
def main():
'''Read AWS requirements from dwh.cfg file'''
config = configparser.ConfigParser()
config.read('dwh.cfg')
'''Connect to the Redshift cluster'''
conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
cur = conn.cursor()
'''Run above functions according to sql_queries.py'''
load_staging_tables(cur, conn)
insert_tables(cur, conn)
conn.close()
if __name__ == "__main__":
main()