3.0 KiB
3.0 KiB
name, description
| name | description |
|---|---|
| dbt_create_transform | Step-by-step instructions for creating a dbt incremental transform model with business logic, joins, and date filters. Use for any dbt + Snowflake project. |
Create a dbt Transform Model
Transform models contain business logic — joins, aggregations, date filters, and derived columns. They are almost always incremental to handle large datasets efficiently.
Steps
1. Name the file
- Pattern:
trf_<domain>_<entity>.sql - Examples:
trf_sales_orders.sql,trf_ar_open_items.sql - Lives in
models/transform/<domain>/
2. Add the config block (top of file)
{{ config(
materialized = 'incremental',
unique_key = ['key_col_1', 'key_col_2'],
on_schema_change = 'sync_all_columns',
incremental_strategy = 'delete+insert'
) }}
unique_key: list of columns that uniquely identify a row.on_schema_change = 'sync_all_columns': automatically adds/removes columns on model changes.
3. (Optional) Add full-refresh protection
{{ sie_dbt_utils.full_refresh_protection() }}
Prevents accidental full-refresh data loss in production. Add on line 1 if using sie_dbt_utils.
4. Define CTEs — one per source model
- Name each CTE after the staging model or business concept it represents.
- Apply tenant/client filters immediately in the CTE (e.g.,
WHERE mandt IN ('022', '100')).
with
orders as (
select * from {{ ref('stg_<system>_<table>') }}
where mandt in ('022', '100') -- replace with your tenant filter
),
customers as (
select * from {{ ref('stg_<system>_kna1') }}
),
5. Write the final SELECT with joins
- Use
UPPER_CASEfor column aliases in transform models (SAP/warehouse convention). - Qualify all columns with CTE alias when joining.
- Use
left join; never barejoin.
6. Add date range filter
- Filter to a rolling window (e.g., last 2 fiscal years) to keep the table small.
- Siemens fiscal year = calendar year + 3 months shift:
where year(dateadd(month, 3, to_date(erdat, 'yyyymmdd'))) >= year(dateadd(year, -2, dateadd(month, 3, getdate())))Adapt the date logic to your project's fiscal/calendar year convention.
7. Add the incremental block
{% if is_incremental() %}
and <date_column> >= (select max(<date_column>) from {{ this }})
{% endif %}
Prefer
sie_dbt_utils.incremental_filter('col')if available — it compiles the MAX watermark at build time.
8. Add schema entry
- File:
models/transform/<domain>/schema.yml - Include model description and
unique+not_nullon the primary key.
Rules
| Rule | Detail |
|---|---|
| Materialization | incremental (set in config block) |
| Column casing | UPPER_CASE aliases for transform/distribute layers |
| Tenant filter | Apply mandt/client filter in each base CTE |
| No hardcoding | Use {{ ref() }} and {{ source() }} only |
| Incremental strategy | delete+insert with unique_key |
Template
See trf_template.sql in this folder.