Files

@gabriel.pereira 6796398924 refactor: restructure monorepo for clean portfolio layout

- Move timesfm-forecast into apps/ directory
- Flatten Udacity portfolio projects from deep URL-encoded paths
  into data-engineering/01-XX numbered directories
- Remove old My-Data-Engineering-Portifolio/ parent directory
- Rewrite root README.md: professional overview with badges,
  project table, and repo structure diagram
- Create data-engineering/README.md with per-project descriptions
- Add README.md for 02-cassandra-modeling (was missing)
- Add README.md for 05-airflow-pipelines (was missing)
- Normalize capstone readme.md -> README.md
- Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat,
  Jupyter checkpoints, IDE dirs; remove uv.lock exclusion
- Add dwh.cfg.example and dl.cfg.example credential templates
- Untrack real credential files (dwh.cfg, dl.cfg)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-03-26 16:48:50 -03:00

2.9 KiB

Raw Blame History

Data Engineering Portfolio

Projects completed during the Udacity Data Engineering Nanodegree — a hands-on journey through relational databases, NoSQL, cloud warehouses, big data processing, pipeline orchestration, and multi-source data lake design.

🎓 View Credential

Projects

01 · Postgres Data Modeling

Stack: Python · PostgreSQL

Design a relational star schema (1 fact + 4 dimensions) for the fictional Sparkify music streaming service. The ETL pipeline reads raw JSON logs and song metadata, then loads them into a normalized database optimized for analytical queries.

Key concepts: Star schema · ETL · psycopg2 · UPSERT logic

02 · Cassandra Data Modeling

Stack: Python · Apache Cassandra

Model a NoSQL schema in Cassandra using a query-first approach. Each table is denormalized to serve exactly one analytical query, demonstrating primary key and clustering column design for distributed databases.

Key concepts: NoSQL denormalization · Partition keys · cassandra-driver

03 · Cloud Data Warehouse

Stack: Python · AWS S3 · AWS Redshift · IAM

Build a cloud ETL pipeline that ingests raw JSON data from S3 into Redshift staging tables, then transforms it into a star schema fact/dimension model for BI analysis.

Key concepts: Redshift COPY command · Staging tables · IAM roles · Columnar storage

⚠️ Requires AWS credentials. Copy dwh.cfg.example → dwh.cfg and fill in your values.

04 · Spark Data Lake

Stack: PySpark · AWS S3 · Parquet

Process large-scale JSON data with Apache Spark using distributed computation. The output is partitioned Parquet files stored back on S3, forming a queryable data lake with fact and dimension tables.

Key concepts: PySpark DataFrames · partitionBy · Parquet format · S3 data lake

⚠️ Requires AWS credentials. Copy dl.cfg.example → dl.cfg and fill in your values.

05 · Airflow Pipelines

Stack: Apache Airflow · AWS S3 · AWS Redshift

Orchestrate an hourly ETL pipeline using a custom Airflow DAG with modular, reusable operators. Includes data quality checks as a final validation step before marking the run complete.

Key concepts: DAG design · Custom operators · StageToRedshiftOperator · Data quality checks

06 · Immigration Capstone