# Automate Pipelines with Apache Airflow

**Udacity Data Engineering Nanodegree — Project 5**

## Overview

Build a production-grade, orchestrated ETL pipeline using **Apache Airflow**. The pipeline runs on an hourly schedule, loading data from AWS S3 into AWS Redshift staging tables, transforming it into a star schema, and validating data quality — all using modular, reusable custom operators.

## Architecture

```
S3 (raw JSON)
    │
    ▼
[StageToRedshiftOperator]    ← Stage events & songs
    │
    ▼
[LoadFactOperator]           ← Load songplays fact table
    │
    ▼
[LoadDimensionOperator] ×4   ← Load users, songs, artists, time
    │
    ▼
[DataQualityOperator]        ← Assert tables are non-empty
```

## DAG

| Property | Value |
|---|---|
| **Schedule** | Hourly (`0 * * * *`) |
| **Start date** | 2019-01-12 |
| **Catchup** | Disabled |

## Custom Operators

| Operator | File | Purpose |
|---|---|---|
| `StageToRedshiftOperator` | `plugins/operators/stage_redshift.py` | COPY JSON from S3 to Redshift staging tables |
| `LoadFactOperator` | `plugins/operators/load_fact.py` | INSERT into fact table from staging |
| `LoadDimensionOperator` | `plugins/operators/load_dimension.py` | INSERT into dimension tables (supports truncate-insert or append) |
| `DataQualityOperator` | `plugins/operators/data_quality.py` | Assert row counts > 0 for all tables |

## Project Structure

```
05-airflow-pipelines/
├── create_tables.sql          # DDL for Redshift tables
├── dags/
│   └── udac_example_dag.py    # Main DAG definition
└── plugins/
    ├── helpers/
    │   └── sql_queries.py     # Shared SQL INSERT statements
    └── operators/
        ├── stage_redshift.py
        ├── load_fact.py
        ├── load_dimension.py
        └── data_quality.py
```

## Key Concepts

- **DAG design** — directed acyclic graphs for workflow orchestration
- **Custom operators** — reusable, parameterized Airflow tasks
- **Modular SQL helpers** — shared query library via `SqlQueries` class
- **Data quality checks** — automated validation at end of every run
- **AWS integration** — S3 `COPY` + Redshift connections via Airflow Connections

## How to Run

1. Set up an Airflow environment with AWS connections configured:
   - `aws_credentials` — IAM access key & secret
   - `redshift` — Redshift cluster connection string

2. Copy DAG and plugins into your Airflow home:
   ```bash
   cp -r dags/ $AIRFLOW_HOME/dags/
   cp -r plugins/ $AIRFLOW_HOME/plugins/
   ```

3. Run `create_tables.sql` against your Redshift cluster to create staging and DW tables.

4. Enable the DAG in the Airflow UI — it will trigger hourly.