# Data Engineering Portfolio

Projects completed during the **Udacity Data Engineering Nanodegree** — a hands-on journey through relational databases, NoSQL, cloud warehouses, big data processing, pipeline orchestration, and multi-source data lake design.

[🎓 View Credential](https://graduation.udacity.com/confirm/43TR6CQZ)

---

## Projects

### 01 · [Postgres Data Modeling](./01-postgres-modeling)

**Stack:** Python · PostgreSQL

Design a relational star schema (1 fact + 4 dimensions) for the fictional Sparkify music streaming service. The ETL pipeline reads raw JSON logs and song metadata, then loads them into a normalized database optimized for analytical queries.

**Key concepts:** Star schema · ETL · `psycopg2` · UPSERT logic

---

### 02 · [Cassandra Data Modeling](./02-cassandra-modeling)

**Stack:** Python · Apache Cassandra

Model a NoSQL schema in Cassandra using a query-first approach. Each table is denormalized to serve exactly one analytical query, demonstrating primary key and clustering column design for distributed databases.

**Key concepts:** NoSQL denormalization · Partition keys · `cassandra-driver`

---

### 03 · [Cloud Data Warehouse](./03-cloud-data-warehouse)

**Stack:** Python · AWS S3 · AWS Redshift · IAM

Build a cloud ETL pipeline that ingests raw JSON data from S3 into Redshift staging tables, then transforms it into a star schema fact/dimension model for BI analysis.

**Key concepts:** Redshift `COPY` command · Staging tables · IAM roles · Columnar storage

> ⚠️ Requires AWS credentials. Copy `dwh.cfg.example` → `dwh.cfg` and fill in your values.

---

### 04 · [Spark Data Lake](./04-spark-data-lake)

**Stack:** PySpark · AWS S3 · Parquet

Process large-scale JSON data with Apache Spark using distributed computation. The output is partitioned Parquet files stored back on S3, forming a queryable data lake with fact and dimension tables.

**Key concepts:** PySpark DataFrames · `partitionBy` · Parquet format · S3 data lake

> ⚠️ Requires AWS credentials. Copy `dl.cfg.example` → `dl.cfg` and fill in your values.

---

### 05 · [Airflow Pipelines](./05-airflow-pipelines)

**Stack:** Apache Airflow · AWS S3 · AWS Redshift

Orchestrate an hourly ETL pipeline using a custom Airflow DAG with modular, reusable operators. Includes data quality checks as a final validation step before marking the run complete.

**Key concepts:** DAG design · Custom operators · `StageToRedshiftOperator` · Data quality checks

---

### 06 · [Immigration Capstone](./06-immigration-capstone)

**Stack:** PySpark · AWS S3 · Parquet

Integrate three heterogeneous data sources — US I94 immigration records, world temperature data, and US city demographics — into a unified star schema data lake. Includes exploratory analysis and formal data quality checks.

**Key concepts:** Multi-source integration · SAS data formats · Data dictionary · Quality assertions