Files

@gabriel.pereira 6796398924 refactor: restructure monorepo for clean portfolio layout

- Move timesfm-forecast into apps/ directory
- Flatten Udacity portfolio projects from deep URL-encoded paths
  into data-engineering/01-XX numbered directories
- Remove old My-Data-Engineering-Portifolio/ parent directory
- Rewrite root README.md: professional overview with badges,
  project table, and repo structure diagram
- Create data-engineering/README.md with per-project descriptions
- Add README.md for 02-cassandra-modeling (was missing)
- Add README.md for 05-airflow-pipelines (was missing)
- Normalize capstone readme.md -> README.md
- Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat,
  Jupyter checkpoints, IDE dirs; remove uv.lock exclusion
- Add dwh.cfg.example and dl.cfg.example credential templates
- Untrack real credential files (dwh.cfg, dl.cfg)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-03-26 16:48:50 -03:00

2.7 KiB

Raw Blame History

Automate Pipelines with Apache Airflow

Udacity Data Engineering Nanodegree — Project 5

Overview

Build a production-grade, orchestrated ETL pipeline using Apache Airflow. The pipeline runs on an hourly schedule, loading data from AWS S3 into AWS Redshift staging tables, transforming it into a star schema, and validating data quality — all using modular, reusable custom operators.

Architecture

S3 (raw JSON)
    │
    ▼
[StageToRedshiftOperator]    ← Stage events & songs
    │
    ▼
[LoadFactOperator]           ← Load songplays fact table
    │
    ▼
[LoadDimensionOperator] ×4   ← Load users, songs, artists, time
    │
    ▼
[DataQualityOperator]        ← Assert tables are non-empty

DAG

Property	Value
Schedule	Hourly (`0 * * * *`)
Start date	2019-01-12
Catchup	Disabled

Custom Operators

Operator	File	Purpose
`StageToRedshiftOperator`	`plugins/operators/stage_redshift.py`	COPY JSON from S3 to Redshift staging tables
`LoadFactOperator`	`plugins/operators/load_fact.py`	INSERT into fact table from staging
`LoadDimensionOperator`	`plugins/operators/load_dimension.py`	INSERT into dimension tables (supports truncate-insert or append)
`DataQualityOperator`	`plugins/operators/data_quality.py`	Assert row counts > 0 for all tables

Project Structure

05-airflow-pipelines/
├── create_tables.sql          # DDL for Redshift tables
├── dags/
│   └── udac_example_dag.py    # Main DAG definition
└── plugins/
    ├── helpers/
    │   └── sql_queries.py     # Shared SQL INSERT statements
    └── operators/
        ├── stage_redshift.py
        ├── load_fact.py
        ├── load_dimension.py
        └── data_quality.py

Key Concepts

DAG design — directed acyclic graphs for workflow orchestration
Custom operators — reusable, parameterized Airflow tasks
Modular SQL helpers — shared query library via SqlQueries class
Data quality checks — automated validation at end of every run
AWS integration — S3 COPY + Redshift connections via Airflow Connections

How to Run

Set up an Airflow environment with AWS connections configured:
- aws_credentials — IAM access key & secret
- redshift — Redshift cluster connection string

Copy DAG and plugins into your Airflow home:

cp -r dags/ $AIRFLOW_HOME/dags/
cp -r plugins/ $AIRFLOW_HOME/plugins/

Run create_tables.sql against your Redshift cluster to create staging and DW tables.
Enable the DAG in the Airflow UI — it will trigger hourly.

2.7 KiB Raw Blame History Unescape Escape