refactor: restructure monorepo for clean portfolio layout
- Move timesfm-forecast into apps/ directory - Flatten Udacity portfolio projects from deep URL-encoded paths into data-engineering/01-XX numbered directories - Remove old My-Data-Engineering-Portifolio/ parent directory - Rewrite root README.md: professional overview with badges, project table, and repo structure diagram - Create data-engineering/README.md with per-project descriptions - Add README.md for 02-cassandra-modeling (was missing) - Add README.md for 05-airflow-pipelines (was missing) - Normalize capstone readme.md -> README.md - Update .gitignore: add *.cfg, *.env, *.zip, *.sas7bdat, Jupyter checkpoints, IDE dirs; remove uv.lock exclusion - Add dwh.cfg.example and dl.cfg.example credential templates - Untrack real credential files (dwh.cfg, dl.cfg) Co-authored-by: Copilot <[email protected]>
This commit is contained in:
1 parent
5c4e6075e1
commit
6796398924
160 files changed
+308
-34
No files matched your search
+17
-3
@@ -20,13 +20,27 @@ build/
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
|
||||
# Streamlit
|
||||
.streamlit/
|
||||
**/.streamlit/
|
||||
|
||||
# Lock files (optional, if not needed)
|
||||
uv.lock
|
||||
# Jupyter
|
||||
.ipynb_checkpoints/
|
||||
**/.ipynb_checkpoints/
|
||||
|
||||
# Data files
|
||||
# Credentials & config with secrets
|
||||
*.cfg
|
||||
*.env
|
||||
.env
|
||||
|
||||
# Large / binary data files
|
||||
*.csv
|
||||
*.zip
|
||||
*.sas7bdat
|
||||
*.parquet
|
||||
@@ -1,10 +0,0 @@
|
||||
[AWS]
|
||||
AWS_ACCESS_KEY_ID=AKIAYZ5XCJA24K6WMN5M
|
||||
AWS_SECRET_ACCESS_KEY=WNhNIEM2WGHlt4E6RjJEAPzt4h2QXNoiL8wuYKso
|
||||
|
||||
[IAM_ROLE]
|
||||
ARN=arn:aws:iam::605437446197:role/myRedshiftRole
|
||||
|
||||
[S3]
|
||||
SOURCE_S3_BUCKET=s3a://gfp-udacity/
|
||||
DEST_S3_BUCKET=s3a://gfp-udacity/
|
||||
@@ -1,29 +1,90 @@
|
||||
### Hey, I'm Gabriel! π
|
||||
### Hey, I'm Gabriel! π
|
||||
|
||||
- π Iβm interested in Data Engineer | Data Scientist | Machine Learning
|
||||
- π± Iβm currently learning ML System Design, AI Engineering, and building scalable data pipelines for real-world corporate environments
|
||||
- π€ Iβm looking to collaborate on open source projects
|
||||
- π« How to reach me: gabriel.pereira@protonmail.com
|
||||
- π Interested in **Data Engineering Β· Data Science Β· ML Engineering**
|
||||
- π± Currently learning ML System Design, AI Engineering, and building scalable data pipelines
|
||||
- π€ Open to collaboration on open source projects
|
||||
- π« Reach me at: gabriel.pereira@protonmail.com
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
||||
[](https://www.linkedin.com/in/gabrielpereira-dev)
|
||||
|
||||
---
|
||||
|
||||
### Skills
|
||||
### Skills
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg"width="36" height="36" alt="SQL" /></a>
|
||||
<a href="https://www.postgresql.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/profileme-dev/refs/heads/main/public/icons/skills/postgresql-colored.svg" width="36" height="36" alt="PostgreSQL" /></a>
|
||||
<a href="https://www.python.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/python-colored.svg" width="36" height="36" alt="Python" /></a>
|
||||
<a href="https://docs.microsoft.com/en-us/cpp/?view=msvc-170" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/cplusplus-colored.svg" width="36" height="36" alt="C++" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="Amazon Web Services" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="RaspiberryPi" /></a>
|
||||
<a href="https://aws.amazon.com" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/aws-colored.svg" width="36" height="36" alt="AWS" /></a>
|
||||
<a href="https://www.docker.com/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/docker-colored.svg" width="36" height="36" alt="Docker" /></a>
|
||||
<a href="https://www.linux.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/linux.svg" width="36" height="36" alt="Linux" /></a>
|
||||
<a href="https://www.raspberrypi.org/" target="_blank" rel="noreferrer"><img src="https://raw.githubusercontent.com/danielcranney/readme-generator/main/public/icons/skills/raspberrypi-colored.svg" width="36" height="36" alt="Raspberry Pi" /></a>
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
This is my workspace, it is a monorepo for my data engineering and data science projects.
|
||||
## π Repository Structure
|
||||
|
||||
This is my personal workspace β a monorepo for data engineering and data science projects.
|
||||
|
||||
```
|
||||
workspace/
|
||||
βββ apps/ # Production applications
|
||||
βββ data-engineering/ # Udacity Data Engineering Nanodegree portfolio
|
||||
βββ skills/ # dbt reference templates & best practices
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## π Apps
|
||||
|
||||
### [`apps/timesfm-forecast`](./apps/timesfm-forecast)
|
||||
|
||||
A production-grade time series forecasting application powered by **Google's TimesFM** model.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Stack** | Python Β· Streamlit Β· PyTorch Β· HuggingFace Transformers |
|
||||
| **Features** | Multi-key forecasting Β· CSV upload/download Β· GPU/CPU support |
|
||||
| **Package manager** | `uv` |
|
||||
|
||||
**Quick start:**
|
||||
```bash
|
||||
cd apps/timesfm-forecast
|
||||
uv venv && source .venv/bin/activate
|
||||
uv pip install -e .
|
||||
timesfm-app
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## π Data Engineering Portfolio
|
||||
|
||||
Udacity Data Engineering Nanodegree projects β a hands-on progression through core DE tools.
|
||||
[π View Credential](https://graduation.udacity.com/confirm/43TR6CQZ)
|
||||
|
||||
| # | Project | Stack | Description |
|
||||
|---|---------|-------|-------------|
|
||||
| 1 | [Postgres Data Modeling](./data-engineering/01-postgres-modeling) | Python Β· PostgreSQL | Star schema design for music streaming analytics |
|
||||
| 2 | [Cassandra Data Modeling](./data-engineering/02-cassandra-modeling) | Python Β· Apache Cassandra | Denormalized NoSQL schema for query-first design |
|
||||
| 3 | [Cloud Data Warehouse](./data-engineering/03-cloud-data-warehouse) | Python Β· AWS S3 Β· Redshift | Cloud ETL pipeline with staging and fact/dim tables |
|
||||
| 4 | [Spark Data Lake](./data-engineering/04-spark-data-lake) | PySpark Β· AWS S3 | Large-scale partitioned processing with Parquet output |
|
||||
| 5 | [Airflow Pipelines](./data-engineering/05-airflow-pipelines) | Apache Airflow Β· Redshift | Orchestrated hourly ETL with modular custom operators |
|
||||
| 6 | [Immigration Capstone](./data-engineering/06-immigration-capstone) | PySpark Β· AWS S3 | Multi-source data lake integrating immigration, weather & demographics |
|
||||
|
||||
See [`data-engineering/README.md`](./data-engineering/README.md) for full details.
|
||||
|
||||
---
|
||||
|
||||
## π οΈ dbt Skills
|
||||
|
||||
The [`skills/`](./skills) folder contains reference templates and best-practice guides for dbt projects:
|
||||
|
||||
- Staging, Transform, and Mart model patterns
|
||||
- Incremental materialization strategies
|
||||
- SQL style guide
|
||||
- YAML schema testing templates
|
||||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
|
500 Internal Server Error
An error occurred: An error occurred Gitea Version: 28.2.0 |