diff --git a/.gitignore b/.gitignore index ce31563..f9bc880 100644 --- a/.gitignore +++ b/.gitignore @@ -20,13 +20,27 @@ build/ # OS files .DS_Store +Thumbs.db + +# IDE +.vscode/ +.idea/ # Streamlit .streamlit/ **/.streamlit/ -# Lock files (optional, if not needed) -uv.lock +# Jupyter +.ipynb_checkpoints/ +**/.ipynb_checkpoints/ -# Data files +# Credentials & config with secrets +*.cfg +*.env +.env + +# Large / binary data files *.csv +*.zip +*.sas7bdat +*.parquet diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/dl.cfg b/My-Data-Engineering-Portifolio/Capstone+Project/dl.cfg deleted file mode 100644 index 828b447..0000000 --- a/My-Data-Engineering-Portifolio/Capstone+Project/dl.cfg +++ /dev/null @@ -1,10 +0,0 @@ -[AWS] -AWS_ACCESS_KEY_ID=AKIAYZ5XCJA24K6WMN5M -AWS_SECRET_ACCESS_KEY=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 - -[IAM_ROLE] -ARN=arn:aws:iam::605437446197:role/myRedshiftRole - -[S3] -SOURCE_S3_BUCKET=s3a://gfp-udacity/ -DEST_S3_BUCKET=s3a://gfp-udacity/ diff --git a/README.md b/README.md index f88493f..d1601b9 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,90 @@ -### Hey, I'm Gabriel! ๐Ÿ‘‹ +### Hey, I'm Gabriel! ๐Ÿ‘‹ -- ๐Ÿ‘€ Iโ€™m interested in Data Engineer | Data Scientist | Machine Learning -- ๐ŸŒฑ Iโ€™m currently learning ML System Design, AI Engineering, and building scalable data pipelines for real-world corporate environments -- ๐Ÿค Iโ€™m looking to collaborate on open source projects -- ๐Ÿ“ซ How to reach me: gabriel.pereira@protonmail.com +- ๐Ÿ‘€ Interested in **Data Engineering ยท Data Science ยท ML Engineering** +- ๐ŸŒฑ Currently learning ML System Design, AI Engineering, and building scalable data pipelines +- ๐Ÿค Open to collaboration on open source projects +- ๐Ÿ“ซ Reach me at: gabriel.pereira@protonmail.com
-[![Linkedin Badge](https://img.shields.io/badge/-LinkedIn-blue?style=flat-square&logo=Linkedin&logoColor=white&link=https://www.linkedin.com/in/gabrielpereira-dev)](https://www.linkedin.com/in/gabrielpereira-dev) +[![LinkedIn](https://img.shields.io/badge/-LinkedIn-blue?style=flat-square&logo=Linkedin&logoColor=white&link=https://www.linkedin.com/in/gabrielpereira-dev)](https://www.linkedin.com/in/gabrielpereira-dev) --- -### Skills +### Skills

- SQL + PostgreSQL Python C++ - Amazon Web Services - Docker - Linux -RaspiberryPi + AWS + Docker + Linux + Raspberry Pi

+
+ --- -This is my workspace, it is a monorepo for my data engineering and data science projects. +## ๐Ÿ“ Repository Structure +This is my personal workspace โ€” a monorepo for data engineering and data science projects. + +``` +workspace/ +โ”œโ”€โ”€ apps/ # Production applications +โ”œโ”€โ”€ data-engineering/ # Udacity Data Engineering Nanodegree portfolio +โ””โ”€โ”€ skills/ # dbt reference templates & best practices +``` + +--- + +## ๐Ÿš€ Apps + +### [`apps/timesfm-forecast`](./apps/timesfm-forecast) + +A production-grade time series forecasting application powered by **Google's TimesFM** model. + +| | | +|---|---| +| **Stack** | Python ยท Streamlit ยท PyTorch ยท HuggingFace Transformers | +| **Features** | Multi-key forecasting ยท CSV upload/download ยท GPU/CPU support | +| **Package manager** | `uv` | + +**Quick start:** +```bash +cd apps/timesfm-forecast +uv venv && source .venv/bin/activate +uv pip install -e . +timesfm-app +``` + +--- + +## ๐Ÿ“š Data Engineering Portfolio + +Udacity Data Engineering Nanodegree projects โ€” a hands-on progression through core DE tools. +[๐ŸŽ“ View Credential](https://graduation.udacity.com/confirm/43TR6CQZ) + +| # | Project | Stack | Description | +|---|---------|-------|-------------| +| 1 | [Postgres Data Modeling](./data-engineering/01-postgres-modeling) | Python ยท PostgreSQL | Star schema design for music streaming analytics | +| 2 | [Cassandra Data Modeling](./data-engineering/02-cassandra-modeling) | Python ยท Apache Cassandra | Denormalized NoSQL schema for query-first design | +| 3 | [Cloud Data Warehouse](./data-engineering/03-cloud-data-warehouse) | Python ยท AWS S3 ยท Redshift | Cloud ETL pipeline with staging and fact/dim tables | +| 4 | [Spark Data Lake](./data-engineering/04-spark-data-lake) | PySpark ยท AWS S3 | Large-scale partitioned processing with Parquet output | +| 5 | [Airflow Pipelines](./data-engineering/05-airflow-pipelines) | Apache Airflow ยท Redshift | Orchestrated hourly ETL with modular custom operators | +| 6 | [Immigration Capstone](./data-engineering/06-immigration-capstone) | PySpark ยท AWS S3 | Multi-source data lake integrating immigration, weather & demographics | + +See [`data-engineering/README.md`](./data-engineering/README.md) for full details. + +--- + +## ๐Ÿ› ๏ธ dbt Skills + +The [`skills/`](./skills) folder contains reference templates and best-practice guides for dbt projects: + +- Staging, Transform, and Mart model patterns +- Incremental materialization strategies +- SQL style guide +- YAML schema testing templates diff --git a/timesfm-forecast/.gitignore b/apps/timesfm-forecast/.gitignore similarity index 100% rename from timesfm-forecast/.gitignore rename to apps/timesfm-forecast/.gitignore diff --git a/timesfm-forecast/LICENSE b/apps/timesfm-forecast/LICENSE similarity index 100% rename from timesfm-forecast/LICENSE rename to apps/timesfm-forecast/LICENSE diff --git a/timesfm-forecast/README.md b/apps/timesfm-forecast/README.md similarity index 100% rename from timesfm-forecast/README.md rename to apps/timesfm-forecast/README.md diff --git a/timesfm-forecast/examples/sample.csv b/apps/timesfm-forecast/examples/sample.csv similarity index 100% rename from timesfm-forecast/examples/sample.csv rename to apps/timesfm-forecast/examples/sample.csv diff --git a/timesfm-forecast/pyproject.toml b/apps/timesfm-forecast/pyproject.toml similarity index 100% rename from timesfm-forecast/pyproject.toml rename to apps/timesfm-forecast/pyproject.toml diff --git a/timesfm-forecast/src/timesfm_app/__init__.py b/apps/timesfm-forecast/src/timesfm_app/__init__.py similarity index 100% rename from timesfm-forecast/src/timesfm_app/__init__.py rename to apps/timesfm-forecast/src/timesfm_app/__init__.py diff --git a/timesfm-forecast/src/timesfm_app/cli.py b/apps/timesfm-forecast/src/timesfm_app/cli.py similarity index 100% rename from timesfm-forecast/src/timesfm_app/cli.py rename to apps/timesfm-forecast/src/timesfm_app/cli.py diff --git a/timesfm-forecast/src/timesfm_app/core.py b/apps/timesfm-forecast/src/timesfm_app/core.py similarity index 100% rename from timesfm-forecast/src/timesfm_app/core.py rename to apps/timesfm-forecast/src/timesfm_app/core.py diff --git a/timesfm-forecast/src/timesfm_app/ui/app.py b/apps/timesfm-forecast/src/timesfm_app/ui/app.py similarity index 100% rename from timesfm-forecast/src/timesfm_app/ui/app.py rename to apps/timesfm-forecast/src/timesfm_app/ui/app.py diff --git a/timesfm-forecast/tests/test_pipeline.py b/apps/timesfm-forecast/tests/test_pipeline.py similarity index 100% rename from timesfm-forecast/tests/test_pipeline.py rename to apps/timesfm-forecast/tests/test_pipeline.py diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/README.md b/data-engineering/01-postgres-modeling/README.md similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/README.md rename to data-engineering/01-postgres-modeling/README.md diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/create_tables.py b/data-engineering/01-postgres-modeling/create_tables.py similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/create_tables.py rename to data-engineering/01-postgres-modeling/create_tables.py diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-01-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-01-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-01-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-01-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-02-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-02-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-02-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-02-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-03-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-03-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-03-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-03-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-04-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-04-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-04-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-04-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-05-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-05-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-05-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-05-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-06-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-06-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-06-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-06-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-07-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-07-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-07-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-07-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-08-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-08-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-08-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-08-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-09-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-09-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-09-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-09-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-10-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-10-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-10-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-10-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-11-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-11-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-11-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-11-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-12-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-12-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-12-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-12-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-13-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-13-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-13-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-13-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-14-events.json b/data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-14-events.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/log_data/2018/11/2018-11-14-events.json rename to data-engineering/01-postgres-modeling/data/log_data/2018/11/2018-11-14-events.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAAW128F429D538.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAAW128F429D538.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAAW128F429D538.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAAW128F429D538.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAABD128F429CF47.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAABD128F429CF47.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAABD128F429CF47.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAABD128F429CF47.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAADZ128F9348C2E.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAADZ128F9348C2E.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAADZ128F9348C2E.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAADZ128F9348C2E.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAEF128F4273421.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAEF128F4273421.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAEF128F4273421.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAEF128F4273421.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAFD128F92F423A.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAFD128F92F423A.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAFD128F92F423A.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAFD128F92F423A.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAMO128F1481E7F.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAMO128F1481E7F.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAMO128F1481E7F.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAMO128F1481E7F.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAMQ128F1460CD3.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAMQ128F1460CD3.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAMQ128F1460CD3.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAMQ128F1460CD3.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAPK128E0786D96.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAPK128E0786D96.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAPK128E0786D96.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAPK128E0786D96.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAARJ128F9320760.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAARJ128F9320760.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAARJ128F9320760.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAARJ128F9320760.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAVG12903CFA543.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAVG12903CFA543.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/A/TRAAAVG12903CFA543.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/A/TRAAAVG12903CFA543.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABCL128F4286650.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABCL128F4286650.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABCL128F4286650.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABCL128F4286650.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABDL12903CAABBA.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABDL12903CAABBA.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABDL12903CAABBA.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABDL12903CAABBA.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABJL12903CDCF1A.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABJL12903CDCF1A.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABJL12903CDCF1A.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABJL12903CDCF1A.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABJV128F1460C49.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABJV128F1460C49.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABJV128F1460C49.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABJV128F1460C49.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABLR128F423B7E3.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABLR128F423B7E3.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABLR128F423B7E3.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABLR128F423B7E3.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABNV128F425CEE1.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABNV128F425CEE1.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABNV128F425CEE1.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABNV128F425CEE1.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABRB128F9306DD5.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABRB128F9306DD5.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABRB128F9306DD5.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABRB128F9306DD5.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABVM128F92CA9DC.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABVM128F92CA9DC.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABVM128F92CA9DC.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABVM128F92CA9DC.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABXG128F9318EBD.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABXG128F9318EBD.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/B/TRAABXG128F9318EBD.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/B/TRAABXG128F9318EBD.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAABYN12903CFD305.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAABYN12903CFD305.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAABYN12903CFD305.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAABYN12903CFD305.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACCG128F92E8A55.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACCG128F92E8A55.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACCG128F92E8A55.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACCG128F92E8A55.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACER128F4290F96.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACER128F4290F96.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACER128F4290F96.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACER128F4290F96.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACFV128F935E50B.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACFV128F935E50B.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACFV128F935E50B.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACFV128F935E50B.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACHN128F1489601.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACHN128F1489601.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACHN128F1489601.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACHN128F1489601.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACIW12903CC0F6D.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACIW12903CC0F6D.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACIW12903CC0F6D.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACIW12903CC0F6D.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACLV128F427E123.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACLV128F427E123.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACLV128F427E123.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACLV128F427E123.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACNS128F14A2DF5.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACNS128F14A2DF5.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACNS128F14A2DF5.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACNS128F14A2DF5.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACPE128F421C1B9.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACPE128F421C1B9.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACPE128F421C1B9.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACPE128F421C1B9.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACQT128F9331780.json b/data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACQT128F9331780.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/A/C/TRAACQT128F9331780.json rename to data-engineering/01-postgres-modeling/data/song_data/A/A/C/TRAACQT128F9331780.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRAACOW128F933E35F.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRAACOW128F933E35F.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRAACOW128F933E35F.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRAACOW128F933E35F.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABACN128F425B784.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABACN128F425B784.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABACN128F425B784.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABACN128F425B784.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAFJ128F42AF24E.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAFJ128F42AF24E.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAFJ128F42AF24E.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAFJ128F42AF24E.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAFP128F931E9A1.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAFP128F931E9A1.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAFP128F931E9A1.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAFP128F931E9A1.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAIO128F42938F9.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAIO128F42938F9.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAIO128F42938F9.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAIO128F42938F9.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABATO128F42627E9.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABATO128F42627E9.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABATO128F42627E9.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABATO128F42627E9.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAVQ12903CBF7E0.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAVQ12903CBF7E0.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAVQ12903CBF7E0.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAVQ12903CBF7E0.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAWW128F4250A31.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAWW128F4250A31.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAWW128F4250A31.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAWW128F4250A31.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXL128F424FC50.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXL128F424FC50.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXL128F424FC50.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXL128F424FC50.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXR128F426515F.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXR128F426515F.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXR128F426515F.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXR128F426515F.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXV128F92F6AE3.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXV128F92F6AE3.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/A/TRABAXV128F92F6AE3.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/A/TRABAXV128F92F6AE3.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBAM128F429D223.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBAM128F429D223.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBAM128F429D223.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBAM128F429D223.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBBV128F42967D7.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBBV128F42967D7.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBBV128F42967D7.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBBV128F42967D7.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBJE12903CDB442.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBJE12903CDB442.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBJE12903CDB442.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBJE12903CDB442.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBKX128F4285205.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBKX128F4285205.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBKX128F4285205.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBKX128F4285205.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBLU128F93349CF.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBLU128F93349CF.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBLU128F93349CF.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBLU128F93349CF.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBNP128F932546F.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBNP128F932546F.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBNP128F932546F.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBNP128F932546F.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBOP128F931B50D.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBOP128F931B50D.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBOP128F931B50D.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBOP128F931B50D.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBOR128F4286200.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBOR128F4286200.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBOR128F4286200.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBOR128F4286200.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBTA128F933D304.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBTA128F933D304.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBTA128F933D304.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBTA128F933D304.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBVJ128F92F7EAA.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBVJ128F92F7EAA.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/B/TRABBVJ128F92F7EAA.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/B/TRABBVJ128F92F7EAA.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCAJ12903CDFCC2.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCAJ12903CDFCC2.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCAJ12903CDFCC2.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCAJ12903CDFCC2.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCEC128F426456E.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCEC128F426456E.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCEC128F426456E.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCEC128F426456E.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCEI128F424C983.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCEI128F424C983.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCEI128F424C983.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCEI128F424C983.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCFL128F149BB0D.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCFL128F149BB0D.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCFL128F149BB0D.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCFL128F149BB0D.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCIX128F4265903.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCIX128F4265903.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCIX128F4265903.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCIX128F4265903.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCKL128F423A778.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCKL128F423A778.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCKL128F423A778.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCKL128F423A778.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCPZ128F4275C32 (1).json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCPZ128F4275C32 (1).json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCPZ128F4275C32 (1).json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCPZ128F4275C32 (1).json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCRU128F423F449.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCRU128F423F449.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCRU128F423F449.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCRU128F423F449.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCTK128F934B224 (1).json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCTK128F934B224 (1).json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCTK128F934B224 (1).json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCTK128F934B224 (1).json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCUQ128E0783E2B.json b/data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCUQ128E0783E2B.json similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/data/song_data/A/B/C/TRABCUQ128E0783E2B.json rename to data-engineering/01-postgres-modeling/data/song_data/A/B/C/TRABCUQ128E0783E2B.json diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/etl.ipynb b/data-engineering/01-postgres-modeling/etl.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/etl.ipynb rename to data-engineering/01-postgres-modeling/etl.ipynb diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/etl.py b/data-engineering/01-postgres-modeling/etl.py similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/etl.py rename to data-engineering/01-postgres-modeling/etl.py diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/sql_queries.py b/data-engineering/01-postgres-modeling/sql_queries.py similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/sql_queries.py rename to data-engineering/01-postgres-modeling/sql_queries.py diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/test.ipynb b/data-engineering/01-postgres-modeling/test.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Postgres/test.ipynb rename to data-engineering/01-postgres-modeling/test.ipynb diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/Project_1B_ Project_Template.ipynb b/data-engineering/02-cassandra-modeling/Project_1B_ Project_Template.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/Project_1B_ Project_Template.ipynb rename to data-engineering/02-cassandra-modeling/Project_1B_ Project_Template.ipynb diff --git a/data-engineering/02-cassandra-modeling/README.md b/data-engineering/02-cassandra-modeling/README.md new file mode 100644 index 0000000..38f9397 --- /dev/null +++ b/data-engineering/02-cassandra-modeling/README.md @@ -0,0 +1,45 @@ +# Data Modeling with Apache Cassandra + +**Udacity Data Engineering Nanodegree โ€” Project 2** + +## Overview + +Model and query a NoSQL database using **Apache Cassandra** for the fictional Sparkify music streaming service. Unlike relational databases, Cassandra requires a _query-first_ design approach: each table is purpose-built to answer one specific analytical question. + +## Dataset + +Raw event data is sourced from CSV log files (`event_data/2018-11-*.csv`) representing daily user activity. These files are pre-processed and merged into a single consolidated dataset (`event_datafile_new.csv`) before loading into Cassandra. + +**Sample fields:** `artist`, `firstName`, `gender`, `itemInSession`, `lastName`, `length`, `level`, `location`, `sessionId`, `song`, `userId` + +## Schema Design + +Three tables are modeled to answer three specific queries: + +| Table | Partition Key | Clustering Columns | Answers | +|---|---|---|---| +| `session_songs` | `sessionId` | `itemInSession` | What song was played in a given session and item? | +| `user_session_songs` | `userId`, `sessionId` | `itemInSession` | What songs did a user listen to in a session? | +| `song_listeners` | `song` | `userId` | Who listened to a specific song? | + +## Key Concepts + +- **Query-first modeling** โ€” schema designed around queries, not entities +- **Denormalization** โ€” data is duplicated across tables to enable fast reads +- **Partition keys** โ€” determine data distribution across nodes +- **Clustering columns** โ€” control sort order within a partition +- **`cassandra-driver`** โ€” Python client for Cassandra + +## How to Run + +1. Install dependencies: + ```bash + pip install cassandra-driver pandas + ``` + +2. Open and run the notebook: + ```bash + jupyter notebook "Project_1B_ Project_Template.ipynb" + ``` + +The notebook walks through preprocessing the CSV data, creating Cassandra tables, inserting records, and running validation queries. diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-01-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-01-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-01-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-01-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-02-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-02-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-02-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-02-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-03-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-03-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-03-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-03-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-04-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-04-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-04-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-04-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-05-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-05-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-05-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-05-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-06-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-06-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-06-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-06-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-07-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-07-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-07-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-07-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-08-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-08-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-08-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-08-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-09-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-09-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-09-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-09-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-10-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-10-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-10-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-10-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-11-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-11-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-11-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-11-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-12-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-12-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-12-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-12-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-13-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-13-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-13-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-13-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-14-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-14-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-14-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-14-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-15-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-15-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-15-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-15-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-16-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-16-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-16-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-16-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-17-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-17-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-17-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-17-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-18-events.csv b/data-engineering/02-cassandra-modeling/event_data/2018-11-18-events.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_data/2018-11-18-events.csv rename to data-engineering/02-cassandra-modeling/event_data/2018-11-18-events.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_datafile_new.csv b/data-engineering/02-cassandra-modeling/event_datafile_new.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/event_datafile_new.csv rename to data-engineering/02-cassandra-modeling/event_datafile_new.csv diff --git a/My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/images/image_event_datafile_new.jpg b/data-engineering/02-cassandra-modeling/images/image_event_datafile_new.jpg similarity index 100% rename from My-Data-Engineering-Portifolio/Data+Modeling/Project+Data+Modeling+with+Apache+Cassandra/images/image_event_datafile_new.jpg rename to data-engineering/02-cassandra-modeling/images/image_event_datafile_new.jpg diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/README.md b/data-engineering/03-cloud-data-warehouse/README.md similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/README.md rename to data-engineering/03-cloud-data-warehouse/README.md diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/create_tables.py b/data-engineering/03-cloud-data-warehouse/create_tables.py similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/create_tables.py rename to data-engineering/03-cloud-data-warehouse/create_tables.py diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/dwh.cfg b/data-engineering/03-cloud-data-warehouse/dwh.cfg.example similarity index 56% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/dwh.cfg rename to data-engineering/03-cloud-data-warehouse/dwh.cfg.example index a76ef7b..3dbe51e 100644 --- a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/dwh.cfg +++ b/data-engineering/03-cloud-data-warehouse/dwh.cfg.example @@ -1,29 +1,29 @@ [AWS] -KEY=AKIAWVLQ3XEKGAJJ6WNT -SECRET=j2CpNC5sY5m1gVt+RmjpZEdgNsn54q8GWuUW+N4F +KEY=YOUR_AWS_ACCESS_KEY_ID +SECRET=YOUR_AWS_SECRET_ACCESS_KEY [CLUSTER] -HOST=redshift-cluster-1.cfw5ezaysudh.us-west-2.redshift.amazonaws.com +HOST=your-cluster.region.redshift.amazonaws.com DB_NAME=dev DB_USER=awsuser -DB_PASSWORD=Yz3bbo4U +DB_PASSWORD=your_password DB_PORT=5439 [IAM_ROLE] -ARN=arn:aws:iam::995187936370:role/myRedshiftRole +ARN=arn:aws:iam::YOUR_ACCOUNT_ID:role/myRedshiftRole [S3] LOG_DATA='s3://udacity-dend/log_data' LOG_JSONPATH='s3://udacity-dend/log_json_path.json' SONG_DATA='s3://udacity-dend/song_data' -[DWH] +[DWH] DWH_CLUSTER_TYPE=multi-node DWH_NUM_NODES=1 DWH_NODE_TYPE=dc2.large DWH_CLUSTER_IDENTIFIER=redshift-cluster-1 DWH_DB=dev DWH_DB_USER=awsuser -DWH_DB_PASSWORD=Yz3bbo4U +DWH_DB_PASSWORD=your_password DWH_PORT=5439 -DWH_IAM_ROLE_NAME=myRedshiftRole \ No newline at end of file +DWH_IAM_ROLE_NAME=myRedshiftRole diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/etl.py b/data-engineering/03-cloud-data-warehouse/etl.py similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/etl.py rename to data-engineering/03-cloud-data-warehouse/etl.py diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/log-data.png b/data-engineering/03-cloud-data-warehouse/log-data.png similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/log-data.png rename to data-engineering/03-cloud-data-warehouse/log-data.png diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/schema_diagram (1).png b/data-engineering/03-cloud-data-warehouse/schema_diagram (1).png similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/schema_diagram (1).png rename to data-engineering/03-cloud-data-warehouse/schema_diagram (1).png diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/sql_queries.py b/data-engineering/03-cloud-data-warehouse/sql_queries.py similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/sql_queries.py rename to data-engineering/03-cloud-data-warehouse/sql_queries.py diff --git a/My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/test.ipynb b/data-engineering/03-cloud-data-warehouse/test.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Cloud+Data+Warehouse/Project+Data+Warehouse/test.ipynb rename to data-engineering/03-cloud-data-warehouse/test.ipynb diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/README.md b/data-engineering/04-spark-data-lake/README.md similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/README.md rename to data-engineering/04-spark-data-lake/README.md diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/data/log-data.zip b/data-engineering/04-spark-data-lake/data/log-data.zip similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/data/log-data.zip rename to data-engineering/04-spark-data-lake/data/log-data.zip diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/data/song-data.zip b/data-engineering/04-spark-data-lake/data/song-data.zip similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/data/song-data.zip rename to data-engineering/04-spark-data-lake/data/song-data.zip diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/dl.cfg b/data-engineering/04-spark-data-lake/dl.cfg similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/dl.cfg rename to data-engineering/04-spark-data-lake/dl.cfg diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/etl.ipynb b/data-engineering/04-spark-data-lake/etl.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/etl.ipynb rename to data-engineering/04-spark-data-lake/etl.ipynb diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/etl.py b/data-engineering/04-spark-data-lake/etl.py similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/etl.py rename to data-engineering/04-spark-data-lake/etl.py diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/fact_dimensional_tables.jpg b/data-engineering/04-spark-data-lake/fact_dimensional_tables.jpg similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/fact_dimensional_tables.jpg rename to data-engineering/04-spark-data-lake/fact_dimensional_tables.jpg diff --git a/My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/log-data.png b/data-engineering/04-spark-data-lake/log-data.png similarity index 100% rename from My-Data-Engineering-Portifolio/Spark+Data+Lakes/Project+Data+Lake/log-data.png rename to data-engineering/04-spark-data-lake/log-data.png diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/.DS_Store b/data-engineering/05-airflow-pipelines/.DS_Store similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/.DS_Store rename to data-engineering/05-airflow-pipelines/.DS_Store diff --git a/data-engineering/05-airflow-pipelines/README.md b/data-engineering/05-airflow-pipelines/README.md new file mode 100644 index 0000000..3bc2d7d --- /dev/null +++ b/data-engineering/05-airflow-pipelines/README.md @@ -0,0 +1,83 @@ +# Automate Pipelines with Apache Airflow + +**Udacity Data Engineering Nanodegree โ€” Project 5** + +## Overview + +Build a production-grade, orchestrated ETL pipeline using **Apache Airflow**. The pipeline runs on an hourly schedule, loading data from AWS S3 into AWS Redshift staging tables, transforming it into a star schema, and validating data quality โ€” all using modular, reusable custom operators. + +## Architecture + +``` +S3 (raw JSON) + โ”‚ + โ–ผ +[StageToRedshiftOperator] โ† Stage events & songs + โ”‚ + โ–ผ +[LoadFactOperator] โ† Load songplays fact table + โ”‚ + โ–ผ +[LoadDimensionOperator] ร—4 โ† Load users, songs, artists, time + โ”‚ + โ–ผ +[DataQualityOperator] โ† Assert tables are non-empty +``` + +## DAG + +| Property | Value | +|---|---| +| **Schedule** | Hourly (`0 * * * *`) | +| **Start date** | 2019-01-12 | +| **Catchup** | Disabled | + +## Custom Operators + +| Operator | File | Purpose | +|---|---|---| +| `StageToRedshiftOperator` | `plugins/operators/stage_redshift.py` | COPY JSON from S3 to Redshift staging tables | +| `LoadFactOperator` | `plugins/operators/load_fact.py` | INSERT into fact table from staging | +| `LoadDimensionOperator` | `plugins/operators/load_dimension.py` | INSERT into dimension tables (supports truncate-insert or append) | +| `DataQualityOperator` | `plugins/operators/data_quality.py` | Assert row counts > 0 for all tables | + +## Project Structure + +``` +05-airflow-pipelines/ +โ”œโ”€โ”€ create_tables.sql # DDL for Redshift tables +โ”œโ”€โ”€ dags/ +โ”‚ โ””โ”€โ”€ udac_example_dag.py # Main DAG definition +โ””โ”€โ”€ plugins/ + โ”œโ”€โ”€ helpers/ + โ”‚ โ””โ”€โ”€ sql_queries.py # Shared SQL INSERT statements + โ””โ”€โ”€ operators/ + โ”œโ”€โ”€ stage_redshift.py + โ”œโ”€โ”€ load_fact.py + โ”œโ”€โ”€ load_dimension.py + โ””โ”€โ”€ data_quality.py +``` + +## Key Concepts + +- **DAG design** โ€” directed acyclic graphs for workflow orchestration +- **Custom operators** โ€” reusable, parameterized Airflow tasks +- **Modular SQL helpers** โ€” shared query library via `SqlQueries` class +- **Data quality checks** โ€” automated validation at end of every run +- **AWS integration** โ€” S3 `COPY` + Redshift connections via Airflow Connections + +## How to Run + +1. Set up an Airflow environment with AWS connections configured: + - `aws_credentials` โ€” IAM access key & secret + - `redshift` โ€” Redshift cluster connection string + +2. Copy DAG and plugins into your Airflow home: + ```bash + cp -r dags/ $AIRFLOW_HOME/dags/ + cp -r plugins/ $AIRFLOW_HOME/plugins/ + ``` + +3. Run `create_tables.sql` against your Redshift cluster to create staging and DW tables. + +4. Enable the DAG in the Airflow UI โ€” it will trigger hourly. diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/create_tables.sql b/data-engineering/05-airflow-pipelines/create_tables.sql similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/create_tables.sql rename to data-engineering/05-airflow-pipelines/create_tables.sql diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/dags/.DS_Store b/data-engineering/05-airflow-pipelines/dags/.DS_Store similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/dags/.DS_Store rename to data-engineering/05-airflow-pipelines/dags/.DS_Store diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/dags/udac_example_dag.py b/data-engineering/05-airflow-pipelines/dags/udac_example_dag.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/dags/udac_example_dag.py rename to data-engineering/05-airflow-pipelines/dags/udac_example_dag.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/.DS_Store b/data-engineering/05-airflow-pipelines/plugins/.DS_Store similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/.DS_Store rename to data-engineering/05-airflow-pipelines/plugins/.DS_Store diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/__init__.py b/data-engineering/05-airflow-pipelines/plugins/__init__.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/__init__.py rename to data-engineering/05-airflow-pipelines/plugins/__init__.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/.DS_Store b/data-engineering/05-airflow-pipelines/plugins/helpers/.DS_Store similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/.DS_Store rename to data-engineering/05-airflow-pipelines/plugins/helpers/.DS_Store diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/__init__.py b/data-engineering/05-airflow-pipelines/plugins/helpers/__init__.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/__init__.py rename to data-engineering/05-airflow-pipelines/plugins/helpers/__init__.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/sql_queries.py b/data-engineering/05-airflow-pipelines/plugins/helpers/sql_queries.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/helpers/sql_queries.py rename to data-engineering/05-airflow-pipelines/plugins/helpers/sql_queries.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/.DS_Store b/data-engineering/05-airflow-pipelines/plugins/operators/.DS_Store similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/.DS_Store rename to data-engineering/05-airflow-pipelines/plugins/operators/.DS_Store diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/__init__.py b/data-engineering/05-airflow-pipelines/plugins/operators/__init__.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/__init__.py rename to data-engineering/05-airflow-pipelines/plugins/operators/__init__.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/data_quality.py b/data-engineering/05-airflow-pipelines/plugins/operators/data_quality.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/data_quality.py rename to data-engineering/05-airflow-pipelines/plugins/operators/data_quality.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/load_dimension.py b/data-engineering/05-airflow-pipelines/plugins/operators/load_dimension.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/load_dimension.py rename to data-engineering/05-airflow-pipelines/plugins/operators/load_dimension.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/load_fact.py b/data-engineering/05-airflow-pipelines/plugins/operators/load_fact.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/load_fact.py rename to data-engineering/05-airflow-pipelines/plugins/operators/load_fact.py diff --git a/My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/stage_redshift.py b/data-engineering/05-airflow-pipelines/plugins/operators/stage_redshift.py similarity index 100% rename from My-Data-Engineering-Portifolio/Automate+Pipelines/plugins/operators/stage_redshift.py rename to data-engineering/05-airflow-pipelines/plugins/operators/stage_redshift.py diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/Capstone Project.ipynb b/data-engineering/06-immigration-capstone/Capstone Project.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/Capstone Project.ipynb rename to data-engineering/06-immigration-capstone/Capstone Project.ipynb diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/I94_SAS_Labels_Descriptions.SAS b/data-engineering/06-immigration-capstone/I94_SAS_Labels_Descriptions.SAS similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/I94_SAS_Labels_Descriptions.SAS rename to data-engineering/06-immigration-capstone/I94_SAS_Labels_Descriptions.SAS diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/Quality-checks.ipynb b/data-engineering/06-immigration-capstone/Quality-checks.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/Quality-checks.ipynb rename to data-engineering/06-immigration-capstone/Quality-checks.ipynb diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/readme.md b/data-engineering/06-immigration-capstone/README.md similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/readme.md rename to data-engineering/06-immigration-capstone/README.md diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/airport-codes_csv.csv b/data-engineering/06-immigration-capstone/airport-codes_csv.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/airport-codes_csv.csv rename to data-engineering/06-immigration-capstone/airport-codes_csv.csv diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/data-model.png b/data-engineering/06-immigration-capstone/data-model.png similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/data-model.png rename to data-engineering/06-immigration-capstone/data-model.png diff --git a/data-engineering/06-immigration-capstone/dl.cfg.example b/data-engineering/06-immigration-capstone/dl.cfg.example new file mode 100644 index 0000000..eab3e8b --- /dev/null +++ b/data-engineering/06-immigration-capstone/dl.cfg.example @@ -0,0 +1,10 @@ +[AWS] +AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID +AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY + +[IAM_ROLE] +ARN=arn:aws:iam::YOUR_ACCOUNT_ID:role/myRedshiftRole + +[S3] +SOURCE_S3_BUCKET=s3a://your-bucket/ +DEST_S3_BUCKET=s3a://your-bucket/ diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/etl-s3-testing.ipynb b/data-engineering/06-immigration-capstone/etl-s3-testing.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/etl-s3-testing.ipynb rename to data-engineering/06-immigration-capstone/etl-s3-testing.ipynb diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/etl.ipynb b/data-engineering/06-immigration-capstone/etl.ipynb similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/etl.ipynb rename to data-engineering/06-immigration-capstone/etl.ipynb diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/etl.py b/data-engineering/06-immigration-capstone/etl.py similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/etl.py rename to data-engineering/06-immigration-capstone/etl.py diff --git a/My-Data-Engineering-Portifolio/Capstone+Project/immigration_data_sample.csv b/data-engineering/06-immigration-capstone/immigration_data_sample.csv similarity index 100% rename from My-Data-Engineering-Portifolio/Capstone+Project/immigration_data_sample.csv rename to data-engineering/06-immigration-capstone/immigration_data_sample.csv diff --git a/My-Data-Engineering-Portifolio/README.md b/data-engineering/PORTFOLIO_ORIGIN.md similarity index 100% rename from My-Data-Engineering-Portifolio/README.md rename to data-engineering/PORTFOLIO_ORIGIN.md diff --git a/data-engineering/README.md b/data-engineering/README.md new file mode 100644 index 0000000..a4d8b32 --- /dev/null +++ b/data-engineering/README.md @@ -0,0 +1,71 @@ +# Data Engineering Portfolio + +Projects completed during the **Udacity Data Engineering Nanodegree** โ€” a hands-on journey through relational databases, NoSQL, cloud warehouses, big data processing, pipeline orchestration, and multi-source data lake design. + +[๐ŸŽ“ View Credential](https://graduation.udacity.com/confirm/43TR6CQZ) + +--- + +## Projects + +### 01 ยท [Postgres Data Modeling](./01-postgres-modeling) + +**Stack:** Python ยท PostgreSQL + +Design a relational star schema (1 fact + 4 dimensions) for the fictional Sparkify music streaming service. The ETL pipeline reads raw JSON logs and song metadata, then loads them into a normalized database optimized for analytical queries. + +**Key concepts:** Star schema ยท ETL ยท `psycopg2` ยท UPSERT logic + +--- + +### 02 ยท [Cassandra Data Modeling](./02-cassandra-modeling) + +**Stack:** Python ยท Apache Cassandra + +Model a NoSQL schema in Cassandra using a query-first approach. Each table is denormalized to serve exactly one analytical query, demonstrating primary key and clustering column design for distributed databases. + +**Key concepts:** NoSQL denormalization ยท Partition keys ยท `cassandra-driver` + +--- + +### 03 ยท [Cloud Data Warehouse](./03-cloud-data-warehouse) + +**Stack:** Python ยท AWS S3 ยท AWS Redshift ยท IAM + +Build a cloud ETL pipeline that ingests raw JSON data from S3 into Redshift staging tables, then transforms it into a star schema fact/dimension model for BI analysis. + +**Key concepts:** Redshift `COPY` command ยท Staging tables ยท IAM roles ยท Columnar storage + +> โš ๏ธ Requires AWS credentials. Copy `dwh.cfg.example` โ†’ `dwh.cfg` and fill in your values. + +--- + +### 04 ยท [Spark Data Lake](./04-spark-data-lake) + +**Stack:** PySpark ยท AWS S3 ยท Parquet + +Process large-scale JSON data with Apache Spark using distributed computation. The output is partitioned Parquet files stored back on S3, forming a queryable data lake with fact and dimension tables. + +**Key concepts:** PySpark DataFrames ยท `partitionBy` ยท Parquet format ยท S3 data lake + +> โš ๏ธ Requires AWS credentials. Copy `dl.cfg.example` โ†’ `dl.cfg` and fill in your values. + +--- + +### 05 ยท [Airflow Pipelines](./05-airflow-pipelines) + +**Stack:** Apache Airflow ยท AWS S3 ยท AWS Redshift + +Orchestrate an hourly ETL pipeline using a custom Airflow DAG with modular, reusable operators. Includes data quality checks as a final validation step before marking the run complete. + +**Key concepts:** DAG design ยท Custom operators ยท `StageToRedshiftOperator` ยท Data quality checks + +--- + +### 06 ยท [Immigration Capstone](./06-immigration-capstone) + +**Stack:** PySpark ยท AWS S3 ยท Parquet + +Integrate three heterogeneous data sources โ€” US I94 immigration records, world temperature data, and US city demographics โ€” into a unified star schema data lake. Includes exploratory analysis and formal data quality checks. + +**Key concepts:** Multi-source integration ยท SAS data formats ยท Data dictionary ยท Quality assertions