refactor: restructure monorepo for clean portfolio layout
- Move timesfm-forecast into apps/ directory - Flatten Udacity portfolio projects from deep URL-encoded paths into data-engineering/01-XX numbered directories - Remove old My-Data-Engineering-Portifolio/ parent directory - Rewrite root README.md: professional overview with badges, project table, and repo structure diagram - Create data-engineering/README.md with per-project descriptions - Add README.md for 02-cassandra-modeling (was missing) - Add README.md for 05-airflow-pipelines (was missing) - Normalize capstone readme.md -> README.md - Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat, Jupyter checkpoints, IDE dirs; remove uv.lock exclusion - Add dwh.cfg.example and dl.cfg.example credential templates - Untrack real credential files (dwh.cfg, dl.cfg) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
83
data-engineering/05-airflow-pipelines/README.md
Normal file
83
data-engineering/05-airflow-pipelines/README.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Automate Pipelines with Apache Airflow
|
||||
|
||||
**Udacity Data Engineering Nanodegree — Project 5**
|
||||
|
||||
## Overview
|
||||
|
||||
Build a production-grade, orchestrated ETL pipeline using **Apache Airflow**. The pipeline runs on an hourly schedule, loading data from AWS S3 into AWS Redshift staging tables, transforming it into a star schema, and validating data quality — all using modular, reusable custom operators.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
S3 (raw JSON)
|
||||
│
|
||||
▼
|
||||
[StageToRedshiftOperator] ← Stage events & songs
|
||||
│
|
||||
▼
|
||||
[LoadFactOperator] ← Load songplays fact table
|
||||
│
|
||||
▼
|
||||
[LoadDimensionOperator] ×4 ← Load users, songs, artists, time
|
||||
│
|
||||
▼
|
||||
[DataQualityOperator] ← Assert tables are non-empty
|
||||
```
|
||||
|
||||
## DAG
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| **Schedule** | Hourly (`0 * * * *`) |
|
||||
| **Start date** | 2019-01-12 |
|
||||
| **Catchup** | Disabled |
|
||||
|
||||
## Custom Operators
|
||||
|
||||
| Operator | File | Purpose |
|
||||
|---|---|---|
|
||||
| `StageToRedshiftOperator` | `plugins/operators/stage_redshift.py` | COPY JSON from S3 to Redshift staging tables |
|
||||
| `LoadFactOperator` | `plugins/operators/load_fact.py` | INSERT into fact table from staging |
|
||||
| `LoadDimensionOperator` | `plugins/operators/load_dimension.py` | INSERT into dimension tables (supports truncate-insert or append) |
|
||||
| `DataQualityOperator` | `plugins/operators/data_quality.py` | Assert row counts > 0 for all tables |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
05-airflow-pipelines/
|
||||
├── create_tables.sql # DDL for Redshift tables
|
||||
├── dags/
|
||||
│ └── udac_example_dag.py # Main DAG definition
|
||||
└── plugins/
|
||||
├── helpers/
|
||||
│ └── sql_queries.py # Shared SQL INSERT statements
|
||||
└── operators/
|
||||
├── stage_redshift.py
|
||||
├── load_fact.py
|
||||
├── load_dimension.py
|
||||
└── data_quality.py
|
||||
```
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **DAG design** — directed acyclic graphs for workflow orchestration
|
||||
- **Custom operators** — reusable, parameterized Airflow tasks
|
||||
- **Modular SQL helpers** — shared query library via `SqlQueries` class
|
||||
- **Data quality checks** — automated validation at end of every run
|
||||
- **AWS integration** — S3 `COPY` + Redshift connections via Airflow Connections
|
||||
|
||||
## How to Run
|
||||
|
||||
1. Set up an Airflow environment with AWS connections configured:
|
||||
- `aws_credentials` — IAM access key & secret
|
||||
- `redshift` — Redshift cluster connection string
|
||||
|
||||
2. Copy DAG and plugins into your Airflow home:
|
||||
```bash
|
||||
cp -r dags/ $AIRFLOW_HOME/dags/
|
||||
cp -r plugins/ $AIRFLOW_HOME/plugins/
|
||||
```
|
||||
|
||||
3. Run `create_tables.sql` against your Redshift cluster to create staging and DW tables.
|
||||
|
||||
4. Enable the DAG in the Airflow UI — it will trigger hourly.
|
||||
Reference in New Issue
Block a user