refactor: restructure monorepo for clean portfolio layout
- Move timesfm-forecast into apps/ directory - Flatten Udacity portfolio projects from deep URL-encoded paths into data-engineering/01-XX numbered directories - Remove old My-Data-Engineering-Portifolio/ parent directory - Rewrite root README.md: professional overview with badges, project table, and repo structure diagram - Create data-engineering/README.md with per-project descriptions - Add README.md for 02-cassandra-modeling (was missing) - Add README.md for 05-airflow-pipelines (was missing) - Normalize capstone readme.md -> README.md - Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat, Jupyter checkpoints, IDE dirs; remove uv.lock exclusion - Add dwh.cfg.example and dl.cfg.example credential templates - Untrack real credential files (dwh.cfg, dl.cfg) Co-authored-by: Copilot <[email protected]>
This commit is contained in:
1 parent
5c4e6075e1
commit
6796398924
160 files changed
+308
-34
No files matched your search
+17
-3
@@ -20,13 +20,27 @@ build/
|
|||||||
|
|
||||||
# OS files
|
# OS files
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
|
||||||
# Streamlit
|
# Streamlit
|
||||||
.streamlit/
|
.streamlit/
|
||||||
**/.streamlit/
|
**/.streamlit/
|
||||||
|
|
||||||
# Lock files (optional, if not needed)
|
# Jupyter
|
||||||
uv.lock
|
.ipynb_checkpoints/
|
||||||
|
**/.ipynb_checkpoints/
|
||||||
|
|
||||||
# Data files
|
# Credentials & config with secrets
|
||||||
|
*.cfg
|
||||||
|
*.env
|
||||||
|
.env
|
||||||
|
|
||||||
|
# Large / binary data files
|
||||||
*.csv
|
*.csv
|
||||||
|
*.zip
|
||||||
|
*.sas7bdat
|
||||||
|
*.parquet
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
[AWS]
|
|
||||||
AWS_ACCESS_KEY_ID=AKIAYZ5XCJA24K6WMN5M
|
|
||||||
AWS_SECRET_ACCESS_KEY=WNhNIEM2WGHlt4E6RjJEAPzt4h2QXNoiL8wuYKso
|
|
||||||
|
|
||||||
[IAM_ROLE]
|
|
||||||
ARN=arn:aws:iam::605437446197:role/myRedshiftRole
|
|
||||||
|
|
||||||
[S3]
|
|
||||||
SOURCE_S3_BUCKET=s3a://gfp-udacity/
|
|
||||||
DEST_S3_BUCKET=s3a://gfp-udacity/
|
|
||||||
@@ -1,29 +1,90 @@
|
|||||||
### Hey, I'm Gabriel! 👋
|
### Hey, I'm Gabriel! 👋
|
||||||
|
|
||||||
- 👀 I’m interested in Data Engineer | Data Scientist | Machine Learning
|
- 👀 Interested in **Data Engineering · Data Science · ML Engineering**
|
||||||
- 🌱 I’m currently learning ML System Design, AI Engineering, and building scalable data pipelines for real-world corporate environments
|
- 🌱 Currently learning ML System Design, AI Engineering, and building scalable data pipelines
|
||||||
- 🤝 I’m looking to collaborate on open source projects
|
- 🤝 Open to collaboration on open source projects
|
||||||
- 📫 How to reach me: gabriel.pereira@protonmail.com
|
- 📫 Reach me at: gabriel.pereira@protonmail.com
|
||||||
|
|
||||||
<div align="center">
|
<div align="center">
|
||||||
|
|
||||||
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Skills
|
### Skills
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg"width="36" height="36" alt="SQL" /></a>
|
<a href="https://www.postgresql.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg" width="36" height="36" alt="PostgreSQL" /></a>
|
||||||
<a href="https://www.python.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/python-colored.svg" width="36" height="36" alt="Python" /></a>
|
<a href="https://www.python.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/python-colored.svg" width="36" height="36" alt="Python" /></a>
|
||||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/cplusplus-colored.svg" width="36" height="36" alt="C++" /></a>
|
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/cplusplus-colored.svg" width="36" height="36" alt="C++" /></a>
|
||||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="Amazon Web Services" /></a>
|
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="AWS" /></a>
|
||||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
<a href="https://www.docker.com/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
||||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
<a href="https://www.linux.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
||||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="RaspiberryPi" /></a>
|
<a href="https://www.raspberrypi.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="Raspberry Pi" /></a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
This is my workspace, it is a monorepo for my data engineering and data science projects.
|
## 📁 Repository Structure
|
||||||
|
|
||||||
|
This is my personal workspace — a monorepo for data engineering and data science projects.
|
||||||
|
|
||||||
|
```
|
||||||
|
workspace/
|
||||||
|
├── apps/ # Production applications
|
||||||
|
├── data-engineering/ # Udacity Data Engineering Nanodegree portfolio
|
||||||
|
└── skills/ # dbt reference templates & best practices
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Apps
|
||||||
|
|
||||||
|
### [`apps/timesfm-forecast`](./apps/timesfm-forecast)
|
||||||
|
|
||||||
|
A production-grade time series forecasting application powered by **Google's TimesFM** model.
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **Stack** | Python · Streamlit · PyTorch · HuggingFace Transformers |
|
||||||
|
| **Features** | Multi-key forecasting · CSV upload/download · GPU/CPU support |
|
||||||
|
| **Package manager** | `uv` |
|
||||||
|
|
||||||
|
**Quick start:**
|
||||||
|
```bash
|
||||||
|
cd apps/timesfm-forecast
|
||||||
|
uv venv && source .venv/bin/activate
|
||||||
|
uv pip install -e .
|
||||||
|
timesfm-app
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Data Engineering Portfolio
|
||||||
|
|
||||||
|
Udacity Data Engineering Nanodegree projects — a hands-on progression through core DE tools.
|
||||||
|
[🎓 View Credential](https://graduation.udacity.com/confirm/43TR6CQZ)
|
||||||
|
|
||||||
|
| # | Project | Stack | Description |
|
||||||
|
|---|---------|-------|-------------|
|
||||||
|
| 1 | [Postgres Data Modeling](./data-engineering/01-postgres-modeling) | Python · PostgreSQL | Star schema design for music streaming analytics |
|
||||||
|
| 2 | [Cassandra Data Modeling](./data-engineering/02-cassandra-modeling) | Python · Apache Cassandra | Denormalized NoSQL schema for query-first design |
|
||||||
|
| 3 | [Cloud Data Warehouse](./data-engineering/03-cloud-data-warehouse) | Python · AWS S3 · Redshift | Cloud ETL pipeline with staging and fact/dim tables |
|
||||||
|
| 4 | [Spark Data Lake](./data-engineering/04-spark-data-lake) | PySpark · AWS S3 | Large-scale partitioned processing with Parquet output |
|
||||||
|
| 5 | [Airflow Pipelines](./data-engineering/05-airflow-pipelines) | Apache Airflow · Redshift | Orchestrated hourly ETL with modular custom operators |
|
||||||
|
| 6 | [Immigration Capstone](./data-engineering/06-immigration-capstone) | PySpark · AWS S3 | Multi-source data lake integrating immigration, weather & demographics |
|
||||||
|
|
||||||
|
See [`data-engineering/README.md`](./data-engineering/README.md) for full details.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ dbt Skills
|
||||||
|
|
||||||
|
The [`skills/`](./skills) folder contains reference templates and best-practice guides for dbt projects:
|
||||||
|
|
||||||
|
- Staging, Transform, and Mart model patterns
|
||||||
|
- Incremental materialization strategies
|
||||||
|
- SQL style guide
|
||||||
|
- YAML schema testing templates
|
||||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
|
500 Internal Server Error
An error occurred: An error occurred Gitea Version: 28.2.0 |