refactor: restructure monorepo for clean portfolio layout
- Move timesfm-forecast into apps/ directory - Flatten Udacity portfolio projects from deep URL-encoded paths into data-engineering/01-XX numbered directories - Remove old My-Data-Engineering-Portifolio/ parent directory - Rewrite root README.md: professional overview with badges, project table, and repo structure diagram - Create data-engineering/README.md with per-project descriptions - Add README.md for 02-cassandra-modeling (was missing) - Add README.md for 05-airflow-pipelines (was missing) - Normalize capstone readme.md -> README.md - Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat, Jupyter checkpoints, IDE dirs; remove uv.lock exclusion - Add dwh.cfg.example and dl.cfg.example credential templates - Untrack real credential files (dwh.cfg, dl.cfg) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@gabriel.pereira
20
.gitignore
vendored
20
.gitignore
vendored
@@ -20,13 +20,27 @@ build/
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
|
||||
# Streamlit
|
||||
.streamlit/
|
||||
**/.streamlit/
|
||||
|
||||
# Lock files (optional, if not needed)
|
||||
uv.lock
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
**/.ipynb_checkpoints/
|
||||
|
||||
# Data files
|
||||
# Credentials & config with secrets
|
||||
*.cfg
|
||||
*.env
|
||||
.env
|
||||
|
||||
# Large / binary data files
|
||||
*.csv
|
||||
*.zip
|
||||
*.sas7bdat
|
||||
*.parquet
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
[AWS]
|
||||
AWS_ACCESS_KEY_ID=AKIAYZ5XCJA24K6WMN5M
|
||||
AWS_SECRET_ACCESS_KEY=WNhNIEM2WGHlt4E6RjJEAPzt4h2QXNoiL8wuYKso
|
||||
|
||||
[IAM_ROLE]
|
||||
ARN=arn:aws:iam::605437446197:role/myRedshiftRole
|
||||
|
||||
[S3]
|
||||
SOURCE_S3_BUCKET=s3a://gfp-udacity/
|
||||
DEST_S3_BUCKET=s3a://gfp-udacity/
|
||||
87
README.md
87
README.md
@@ -1,29 +1,90 @@
|
||||
### Hey, I'm Gabriel! 👋
|
||||
### Hey, I'm Gabriel! 👋
|
||||
|
||||
- 👀 I’m interested in Data Engineer | Data Scientist | Machine Learning
|
||||
- 🌱 I’m currently learning ML System Design, AI Engineering, and building scalable data pipelines for real-world corporate environments
|
||||
- 🤝 I’m looking to collaborate on open source projects
|
||||
- 📫 How to reach me: gabriel.pereira@protonmail.com
|
||||
- 👀 Interested in **Data Engineering · Data Science · ML Engineering**
|
||||
- 🌱 Currently learning ML System Design, AI Engineering, and building scalable data pipelines
|
||||
- 🤝 Open to collaboration on open source projects
|
||||
- 📫 Reach me at: gabriel.pereira@protonmail.com
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
||||
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
||||
|
||||
---
|
||||
|
||||
### Skills
|
||||
### Skills
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg"width="36" height="36" alt="SQL" /></a>
|
||||
<a href="https://www.postgresql.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg" width="36" height="36" alt="PostgreSQL" /></a>
|
||||
<a href="https://www.python.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/python-colored.svg" width="36" height="36" alt="Python" /></a>
|
||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/cplusplus-colored.svg" width="36" height="36" alt="C++" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="Amazon Web Services" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="RaspiberryPi" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="AWS" /></a>
|
||||
<a href="https://www.docker.com/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
||||
<a href="https://www.linux.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
||||
<a href="https://www.raspberrypi.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="Raspberry Pi" /></a>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
This is my workspace, it is a monorepo for my data engineering and data science projects.
|
||||
## 📁 Repository Structure
|
||||
|
||||
This is my personal workspace — a monorepo for data engineering and data science projects.
|
||||
|
||||
```
|
||||
workspace/
|
||||
├── apps/ # Production applications
|
||||
├── data-engineering/ # Udacity Data Engineering Nanodegree portfolio
|
||||
└── skills/ # dbt reference templates & best practices
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Apps
|
||||
|
||||
### [`apps/timesfm-forecast`](./apps/timesfm-forecast)
|
||||
|
||||
A production-grade time series forecasting application powered by **Google's TimesFM** model.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Stack** | Python · Streamlit · PyTorch · HuggingFace Transformers |
|
||||
| **Features** | Multi-key forecasting · CSV upload/download · GPU/CPU support |
|
||||
| **Package manager** | `uv` |
|
||||
|
||||
**Quick start:**
|
||||
```bash
|
||||
cd apps/timesfm-forecast
|
||||
uv venv && source .venv/bin/activate
|
||||
uv pip install -e .
|
||||
timesfm-app
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Data Engineering Portfolio
|
||||
|
||||
Udacity Data Engineering Nanodegree projects — a hands-on progression through core DE tools.
|
||||
[🎓 View Credential](https://graduation.udacity.com/confirm/43TR6CQZ)
|
||||
|
||||
| # | Project | Stack | Description |
|
||||
|---|---------|-------|-------------|
|
||||
| 1 | [Postgres Data Modeling](./data-engineering/01-postgres-modeling) | Python · PostgreSQL | Star schema design for music streaming analytics |
|
||||
| 2 | [Cassandra Data Modeling](./data-engineering/02-cassandra-modeling) | Python · Apache Cassandra | Denormalized NoSQL schema for query-first design |
|
||||
| 3 | [Cloud Data Warehouse](./data-engineering/03-cloud-data-warehouse) | Python · AWS S3 · Redshift | Cloud ETL pipeline with staging and fact/dim tables |
|
||||
| 4 | [Spark Data Lake](./data-engineering/04-spark-data-lake) | PySpark · AWS S3 | Large-scale partitioned processing with Parquet output |
|
||||
| 5 | [Airflow Pipelines](./data-engineering/05-airflow-pipelines) | Apache Airflow · Redshift | Orchestrated hourly ETL with modular custom operators |
|
||||
| 6 | [Immigration Capstone](./data-engineering/06-immigration-capstone) | PySpark · AWS S3 | Multi-source data lake integrating immigration, weather & demographics |
|
||||
|
||||
See [`data-engineering/README.md`](./data-engineering/README.md) for full details.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ dbt Skills
|
||||
|
||||
The [`skills/`](./skills) folder contains reference templates and best-practice guides for dbt projects:
|
||||
|
||||
- Staging, Transform, and Mart model patterns
|
||||
- Incremental materialization strategies
|
||||
- SQL style guide
|
||||
- YAML schema testing templates
|
||||
|
||||
45
data-engineering/02-cassandra-modeling/README.md
Normal file
45
data-engineering/02-cassandra-modeling/README.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# Data Modeling with Apache Cassandra
|
||||
|
||||
**Udacity Data Engineering Nanodegree — Project 2**
|
||||
|
||||
## Overview
|
||||
|
||||
Model and query a NoSQL database using **Apache Cassandra** for the fictional Sparkify music streaming service. Unlike relational databases, Cassandra requires a _query-first_ design approach: each table is purpose-built to answer one specific analytical question.
|
||||
|
||||
## Dataset
|
||||
|
||||
Raw event data is sourced from CSV log files (`event_data/2018-11-*.csv`) representing daily user activity. These files are pre-processed and merged into a single consolidated dataset (`event_datafile_new.csv`) before loading into Cassandra.
|
||||
|
||||
**Sample fields:** `artist`, `firstName`, `gender`, `itemInSession`, `lastName`, `length`, `level`, `location`, `sessionId`, `song`, `userId`
|
||||
|
||||
## Schema Design
|
||||
|
||||
Three tables are modeled to answer three specific queries:
|
||||
|
||||
| Table | Partition Key | Clustering Columns | Answers |
|
||||
|---|---|---|---|
|
||||
| `session_songs` | `sessionId` | `itemInSession` | What song was played in a given session and item? |
|
||||
| `user_session_songs` | `userId`, `sessionId` | `itemInSession` | What songs did a user listen to in a session? |
|
||||
| `song_listeners` | `song` | `userId` | Who listened to a specific song? |
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Query-first modeling** — schema designed around queries, not entities
|
||||
- **Denormalization** — data is duplicated across tables to enable fast reads
|
||||
- **Partition keys** — determine data distribution across nodes
|
||||
- **Clustering columns** — control sort order within a partition
|
||||
- **`cassandra-driver`** — Python client for Cassandra
|
||||
|
||||
## How to Run
|
||||
|
||||
1. Install dependencies:
|
||||
```bash
|
||||
pip install cassandra-driver pandas
|
||||
```
|
||||
|
||||
2. Open and run the notebook:
|
||||
```bash
|
||||
jupyter notebook "Project_1B_ Project_Template.ipynb"
|
||||
```
|
||||
|
||||
The notebook walks through preprocessing the CSV data, creating Cassandra tables, inserting records, and running validation queries.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user