95 lines
3.0 KiB
Markdown
95 lines
3.0 KiB
Markdown
---
|
|
name: dbt_create_transform
|
|
description: "Step-by-step instructions for creating a dbt incremental transform model with business logic, joins, and date filters. Use for any dbt + Snowflake project."
|
|
---
|
|
|
|
# Create a dbt Transform Model
|
|
|
|
Transform models contain **business logic** — joins, aggregations, date filters, and derived columns. They are almost always **incremental** to handle large datasets efficiently.
|
|
|
|
---
|
|
|
|
## Steps
|
|
|
|
**1. Name the file**
|
|
- Pattern: `trf_<domain>_<entity>.sql`
|
|
- Examples: `trf_sales_orders.sql`, `trf_ar_open_items.sql`
|
|
- Lives in `models/transform/<domain>/`
|
|
|
|
**2. Add the config block (top of file)**
|
|
```sql
|
|
{{ config(
|
|
materialized = 'incremental',
|
|
unique_key = ['key_col_1', 'key_col_2'],
|
|
on_schema_change = 'sync_all_columns',
|
|
incremental_strategy = 'delete+insert'
|
|
) }}
|
|
```
|
|
- `unique_key`: list of columns that uniquely identify a row.
|
|
- `on_schema_change = 'sync_all_columns'`: automatically adds/removes columns on model changes.
|
|
|
|
**3. (Optional) Add full-refresh protection**
|
|
```sql
|
|
{{ sie_dbt_utils.full_refresh_protection() }}
|
|
```
|
|
> Prevents accidental full-refresh data loss in production. Add on line 1 if using sie_dbt_utils.
|
|
|
|
**4. Define CTEs — one per source model**
|
|
- Name each CTE after the staging model or business concept it represents.
|
|
- Apply tenant/client filters immediately in the CTE (e.g., `WHERE mandt IN ('022', '100')`).
|
|
|
|
```sql
|
|
with
|
|
orders as (
|
|
select * from {{ ref('stg_<system>_<table>') }}
|
|
where mandt in ('022', '100') -- replace with your tenant filter
|
|
),
|
|
customers as (
|
|
select * from {{ ref('stg_<system>_kna1') }}
|
|
),
|
|
```
|
|
|
|
**5. Write the final SELECT with joins**
|
|
- Use `UPPER_CASE` for column aliases in transform models (SAP/warehouse convention).
|
|
- Qualify all columns with CTE alias when joining.
|
|
- Use `left join`; never bare `join`.
|
|
|
|
**6. Add date range filter**
|
|
- Filter to a rolling window (e.g., last 2 fiscal years) to keep the table small.
|
|
- Siemens fiscal year = calendar year + 3 months shift:
|
|
```sql
|
|
where year(dateadd(month, 3, to_date(erdat, 'yyyymmdd')))
|
|
>= year(dateadd(year, -2, dateadd(month, 3, getdate())))
|
|
```
|
|
> Adapt the date logic to your project's fiscal/calendar year convention.
|
|
|
|
**7. Add the incremental block**
|
|
```sql
|
|
{% if is_incremental() %}
|
|
and <date_column> >= (select max(<date_column>) from {{ this }})
|
|
{% endif %}
|
|
```
|
|
> Prefer `sie_dbt_utils.incremental_filter('col')` if available — it compiles the MAX watermark at build time.
|
|
|
|
**8. Add schema entry**
|
|
- File: `models/transform/<domain>/schema.yml`
|
|
- Include model description and `unique` + `not_null` on the primary key.
|
|
|
|
---
|
|
|
|
## Rules
|
|
|
|
| Rule | Detail |
|
|
|------|--------|
|
|
| Materialization | `incremental` (set in config block) |
|
|
| Column casing | `UPPER_CASE` aliases for transform/distribute layers |
|
|
| Tenant filter | Apply `mandt`/client filter in each base CTE |
|
|
| No hardcoding | Use `{{ ref() }}` and `{{ source() }}` only |
|
|
| Incremental strategy | `delete+insert` with `unique_key` |
|
|
|
|
---
|
|
|
|
## Template
|
|
|
|
See `trf_template.sql` in this folder.
|