refactor: restructure monorepo for clean portfolio layout

- Move timesfm-forecast into apps/ directory
- Flatten Udacity portfolio projects from deep URL-encoded paths
  into data-engineering/01-XX numbered directories
- Remove old My-Data-Engineering-Portifolio/ parent directory
- Rewrite root README.md: professional overview with badges,
  project table, and repo structure diagram
- Create data-engineering/README.md with per-project descriptions
- Add README.md for 02-cassandra-modeling (was missing)
- Add README.md for 05-airflow-pipelines (was missing)
- Normalize capstone readme.md -> README.md
- Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat,
  Jupyter checkpoints, IDE dirs; remove uv.lock exclusion
- Add dwh.cfg.example and dl.cfg.example credential templates
- Untrack real credential files (dwh.cfg, dl.cfg)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@gabriel.pereira
2026-03-26 16:48:50 -03:00
parent 5c4e6075e1
commit 6796398924
160 changed files with 308 additions and 34 deletions

View File

@@ -0,0 +1,71 @@
# Data Engineering Portfolio
Projects completed during the **Udacity Data Engineering Nanodegree** — a hands-on journey through relational databases, NoSQL, cloud warehouses, big data processing, pipeline orchestration, and multi-source data lake design.
[🎓 View Credential](https://graduation.udacity.com/confirm/43TR6CQZ)
---
## Projects
### 01 · [Postgres Data Modeling](./01-postgres-modeling)
**Stack:** Python · PostgreSQL
Design a relational star schema (1 fact + 4 dimensions) for the fictional Sparkify music streaming service. The ETL pipeline reads raw JSON logs and song metadata, then loads them into a normalized database optimized for analytical queries.
**Key concepts:** Star schema · ETL · `psycopg2` · UPSERT logic
---
### 02 · [Cassandra Data Modeling](./02-cassandra-modeling)
**Stack:** Python · Apache Cassandra
Model a NoSQL schema in Cassandra using a query-first approach. Each table is denormalized to serve exactly one analytical query, demonstrating primary key and clustering column design for distributed databases.
**Key concepts:** NoSQL denormalization · Partition keys · `cassandra-driver`
---
### 03 · [Cloud Data Warehouse](./03-cloud-data-warehouse)
**Stack:** Python · AWS S3 · AWS Redshift · IAM
Build a cloud ETL pipeline that ingests raw JSON data from S3 into Redshift staging tables, then transforms it into a star schema fact/dimension model for BI analysis.
**Key concepts:** Redshift `COPY` command · Staging tables · IAM roles · Columnar storage
> ⚠️ Requires AWS credentials. Copy `dwh.cfg.example` → `dwh.cfg` and fill in your values.
---
### 04 · [Spark Data Lake](./04-spark-data-lake)
**Stack:** PySpark · AWS S3 · Parquet
Process large-scale JSON data with Apache Spark using distributed computation. The output is partitioned Parquet files stored back on S3, forming a queryable data lake with fact and dimension tables.
**Key concepts:** PySpark DataFrames · `partitionBy` · Parquet format · S3 data lake
> ⚠️ Requires AWS credentials. Copy `dl.cfg.example` → `dl.cfg` and fill in your values.
---
### 05 · [Airflow Pipelines](./05-airflow-pipelines)
**Stack:** Apache Airflow · AWS S3 · AWS Redshift
Orchestrate an hourly ETL pipeline using a custom Airflow DAG with modular, reusable operators. Includes data quality checks as a final validation step before marking the run complete.
**Key concepts:** DAG design · Custom operators · `StageToRedshiftOperator` · Data quality checks
---
### 06 · [Immigration Capstone](./06-immigration-capstone)
**Stack:** PySpark · AWS S3 · Parquet
Integrate three heterogeneous data sources — US I94 immigration records, world temperature data, and US city demographics — into a unified star schema data lake. Includes exploratory analysis and formal data quality checks.
**Key concepts:** Multi-source integration · SAS data formats · Data dictionary · Quality assertions