Compare commits

...

10 Commits

Author SHA1 Message Date
73a2ed8afb sync local 2026-09-12 09:31:04 -03:00
@gabriel.pereira
9d0a58735c add missing LLM context 2026-09-11 11:57:52 -03:00
@gabriel.pereira
52d3e870d2 docs: simplify anonymized examples
Use clean generic names instead of repeated redaction placeholders.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 11:53:37 -03:00
@gabriel.pereira
5c01a80f1a chore: ignore interactive diagram sources
Keep diagram source files local while publishing the static PDF.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 11:39:44 -03:00
@gabriel.pereira
bd06bf51e6 fix: remove PDF browser headers
Regenerate PDFs through the print API without dates, file URLs, or browser metadata.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 11:34:57 -03:00
@gabriel.pereira
e45660d869 docs: remove PDF print metadata
Regenerate case-study PDFs without browser timestamps, file paths, or print headers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 11:25:43 -03:00
@gabriel.pereira
f4dcd73676 review docs 2026-09-11 11:18:14 -03:00
@gabriel.pereira
cf194dc908 remove old files 2026-09-11 11:02:55 -03:00
@gabriel.pereira
c08a667ce5 docs: improve architecture PDF layout
Print only the focused architecture view with a single-page layout.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 11:01:28 -03:00
@gabriel.pereira
9a1963dca2 docs: add architecture diagram PDF
Provide a static version of the architecture for repository viewers.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-09-11 10:59:49 -03:00
9 changed files with 49 additions and 476 deletions

2
.gitignore vendored
View File

@@ -2,3 +2,5 @@ local/
case_study.css case_study.css
case_study_one_page.html case_study_one_page.html
case_study_one_page.md case_study_one_page.md
architecture_dag.md
architecture_diagram.html

View File

@@ -1,6 +1,6 @@
# Talk-to-Data: Semantic Analytics for Enterprise Quotations # Talk-to-Data: Semantic Analytics for Enterprise Quotations
**A POC case study in governance-first semantic layer design: SAP → Snowflake → Cortex → Chatbot.** **A PoC case study in governance-first semantic layer design: SAP → Snowflake → Cortex Analytics (LLM) → Chatbot.**
--- ---
@@ -25,44 +25,44 @@ Sales and operations teams across multiple regions handle thousands of quote req
``` ```
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────┐
│ SAP ERP (Sales, Pricing, Discounts) │ │ SAP ERP (Sales, Pricing, Discounts) │
│ Updated daily via batch extract (05:00 UTC) │ │ Updated daily via batch extract (05:00 UTC) │
└────────────────────────┬────────────────────────────────────┘ └────────────────────────┬────────────────────────────────────┘
│ │
Daily ELT Job Daily ELT Job
│ │
┌────────────────────────▼────────────────────────────────────┐ ┌────────────────────────▼────────────────────────────────────┐
│ Snowflake Raw Layer │ │ Snowflake Raw Layer │
│ - raw.quotations (document, line item, amount, ...) │ │ - raw.quotations (document, line item, amount, ...) │
│ - raw.discount_conditions (discount tiers, policies) │ │ - raw.discount_conditions (discount tiers, policies) │
│ - raw.customer_master (customer codes, agreements) │ │ - raw.customer_master (customer codes, agreements) │
└────────────────────────┬────────────────────────────────────┘ └────────────────────────┬────────────────────────────────────┘
│ │
dbt Staging dbt Staging
│ │
┌────────────────────────▼────────────────────────────────────┐ ┌────────────────────────▼────────────────────────────────────┐
│ Staging Layer (STAGING schema) │ │ Staging Layer (STAGING schema) │
│ - Column mapping, type casting, null handling │ │ - Column mapping, type casting, null handling │
│ - No business logic, just data preparation │ │ - No business logic, just data preparation │
└────────────────────────┬────────────────────────────────────┘ └────────────────────────┬────────────────────────────────────┘
│ │
dbt Transforms dbt Transforms
│ │
┌────────────────────────▼────────────────────────────────────┐ ┌────────────────────────▼────────────────────────────────────┐
│ Transform Layer (TRANSFORM schema) │ │ Transform Layer (TRANSFORM schema) │
│ - Join discount conditions, policies, agreements │ │ - Join discount conditions, policies, agreements │
│ - Calculate: discount tiers, totals, exceptions │ │ - Calculate: discount tiers, totals, exceptions │
│ - Apply: governance flags, audit columns │ │ - Apply: governance flags, audit columns │
│ - Models: trf_quotation, trf_chatbot_quotation │ │ - Models: trf_quotation, trf_chatbot_quotation │
└────────────────────────┬────────────────────────────────────┘ └────────────────────────┬────────────────────────────────────┘
│ │
Semantic Views (DDL) Semantic Views (DDL)
│ │
┌────────────────────────▼────────────────────────────────────┐ ┌────────────────────────▼────────────────────────────────────┐
│ Semantic Layer (DISTRIBUTE_DDL schema) │ │ Semantic Layer (DISTRIBUTE_DDL schema) │
│ - sv_quotation: Full BI consumption │ │ - sv_quotation: Full BI consumption │
│ - sv_chatbot_quotation: LLM-optimized dimensions │ │ - sv_chatbot_quotation: LLM-optimized dimensions │
│ - Exposes: business logic (discounts, policies) │ │ - Exposes: business logic (discounts, policies) │
│ - Hides: implementation details (temp tables, keys) │ │ - Hides: implementation details (temp tables, keys) │
└────────────────────────┬────────────────────────────────────┘ └────────────────────────┬────────────────────────────────────┘
│ │
Cortex Analytics Guardrails Cortex Analytics Guardrails
@@ -76,8 +76,8 @@ Sales and operations teams across multiple regions handle thousands of quote req
│ - Visual analysis │ │ - Quote Q&A │ │ - Visual analysis │ │ - Quote Q&A │
│ - Sales reporting │ │ - Prompt engineering │ │ - Sales reporting │ │ - Prompt engineering │
│ - Ops monitoring │ │ - Real-time answers │ │ - Ops monitoring │ │ - Real-time answers │
└───────────┬──────────┘ └──────────┬────────────┘ └───────────┬───── ─────┘ └──────────┬────────────┘
│ │ │ │
│ <1 minute turnaround │ <1 minute turnaround
│ │
End Users (Sales, Ops) End Users (Sales, Ops)
@@ -178,7 +178,7 @@ Governance is not a checklist; it's embedded in the data layer.
--- ---
## Results: From POC to Impact ## Results: From PoC to Impact
### Estimated Impact (Based on User Interviews) ### Estimated Impact (Based on User Interviews)
- **Quote turnaround**: 5-30 min (email + manual lookup) → <1 min (chatbot query) - **Quote turnaround**: 5-30 min (email + manual lookup) → <1 min (chatbot query)
@@ -186,7 +186,7 @@ Governance is not a checklist; it's embedded in the data layer.
- Pilots with 5 sales reps: ~50+ hours/week freed (estimated) - Pilots with 5 sales reps: ~50+ hours/week freed (estimated)
- Extrapolation: 30+ sales team × 5 hours/week = 150+ hours recovered org-wide - Extrapolation: 30+ sales team × 5 hours/week = 150+ hours recovered org-wide
- **Error reduction**: Discount exceptions caught by policy guardrails (vs. manually reviewed) - **Error reduction**: Discount exceptions caught by policy guardrails (vs. manually reviewed)
- **Team velocity**: POC shipped in weeks (not months), governance patterns established for scale - **Team velocity**: PoC shipped in weeks (not months), governance patterns established for scale
### Why This Matters for Delivery Managers ### Why This Matters for Delivery Managers
1. **Governance-first approach**: We didn't build a prototype and hope for policy later. Guardrails were baked in from day one. 1. **Governance-first approach**: We didn't build a prototype and hope for policy later. Guardrails were baked in from day one.
@@ -196,29 +196,6 @@ Governance is not a checklist; it's embedded in the data layer.
--- ---
## How to Use This Repo
### Deliverables
- **`dbt/`**: Anonymized dbt models (staging, transforms, semantic views)
- `models/staging/stg_*.sql` — Raw layer transformations
- `models/transform/trf_*.sql` — Business logic
- `models/distribute_ddl/sv_*.yml` — Semantic views with synonyms
- **`cortex/`**: Governance config (information protection, access control, audit rules)
- `cortex_governance_config.yaml` — Guardrails, freshness contracts, LLM policies
- **`docs/`**: Architecture diagrams, decision logs
- `architecture_dag.md` — Data flow Mermaid diagram
- `decisions.md` — Architecture decision records (ADRs)
- **`README.md`**: This file (the narrative for interviewers/stakeholders)
### How to Adapt This to Your Domain
1. **Replace discount logic** with your domain (e.g., pricing tiers, contract terms, approval workflows)
2. **Adjust freshness** (once-daily to hourly/real-time) based on your use case
3. **Modify semantic dimensions** to match your business terminology
4. **Update Cortex policies** for your personal-information and governance rules
5. **Test with a small pilot** (5-10 users) before org-wide rollout
---
## Lessons Learned ## Lessons Learned
### What Worked ### What Worked
@@ -231,10 +208,6 @@ Governance is not a checklist; it's embedded in the data layer.
✅ **dbt for team alignment**: SQL-first + schema docs meant the data team and business stakeholders spoke the same language. Onboarding new people was fast. ✅ **dbt for team alignment**: SQL-first + schema docs meant the data team and business stakeholders spoke the same language. Onboarding new people was fast.
### What We'd Change at Scale ### What We'd Change at Scale
⚠️ **Real-time discount updates**: If pilot shows sales need <1hr freshness on policy changes, escalate to hourly ELT.
⚠️ **Multi-region governance**: Current config is single-region (BR). If expanding to US/CA, add region-specific discount tiers and approval workflows.
⚠️ **Cortex cost**: Monitor Cortex token usage as chatbot volume scales. Current config has cost controls (monthly budget, per-query max tokens); may need aggressive pruning at scale. ⚠️ **Cortex cost**: Monitor Cortex token usage as chatbot volume scales. Current config has cost controls (monthly budget, per-query max tokens); may need aggressive pruning at scale.
⚠️ **Semantic view complexity**: If adding more dimensions (15→50), consider splitting into focused semantic views (e.g., `sv_quote_discounts`, `sv_quote_compliance`) to keep queries fast. ⚠️ **Semantic view complexity**: If adding more dimensions (15→50), consider splitting into focused semantic views (e.g., `sv_quote_discounts`, `sv_quote_compliance`) to keep queries fast.
@@ -263,37 +236,14 @@ A: Yes. Adapt the dbt dialect (BigQuery: `jinja-sql`, Redshift: `redshift` profi
**Q: Is once-daily refresh really enough?** **Q: Is once-daily refresh really enough?**
A: For quote research, yes. If you need real-time pricing updates, escalate to hourly ELT. The architecture supports it; just change the schedule in your orchestrator. A: For quote research, yes. If you need real-time pricing updates, escalate to hourly ELT. The architecture supports it; just change the schedule in your orchestrator.
**Q: How do I handle discount exceptions?**
A: Cortex policy flags them and escalates to a human review queue (max 10/hour). Ops team approves or rejects in Snowflake; bot learns the decision for future similar quotes.
**Q: What if I need to scale to 500+ users?** **Q: What if I need to scale to 500+ users?**
A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same. A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same. The real issue comes when you need RLS requirements — this will require additional tools or methods beyond Snowflake RBAC.
**Q: Can I export this as a Snowflake Native App for partners?** **Q: Can I export this as a Snowflake Native App for partners?**
A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data. A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data. Cortex Analyst can be seamlessly integrated into any application using REST API.
--- ---
## Next Steps ## Context
1. **Expand pilot**: Rollout to 30+ sales team over 4 weeks This case study was prepared as a technical exploration demonstrating governance-first semantic layer design for enterprise AI enablement. It reflects delivery patterns developed in a global industrial environment and adapted for privacy-safe public sharing.
2. **Measure impact**: Track quote turnaround times, bot usage, escalation rates
3. **Iterate on policies**: Refine discount rules and information-protection controls based on pilot feedback
4. **Add new domains**: Reuse the semantic layer pattern for contracts, serviceability, pricing
5. **Scale LLM**: Move from proof-of-concept to production volume (monitor cost, latency)
---
## Contact & Questions
This case study demonstrates:
- Governance-first architecture for enterprise AI
- Semantic layer design (SAP → dbt → Snowflake → LLM)
- Operationalizing guardrails and audit trails
- Shipping production-ready POCs in weeks
For questions about the design decisions, data flow, or how to adapt this to your domain, reach out.
---
**Co-authored by Copilot**

View File

@@ -1,59 +0,0 @@
# Quotation Semantic Layer - Data Flow DAG
# SAP Source -> Snowflake -> dbt Transforms -> Semantic Views -> Cortex -> Chatbot Output
graph LR
SAP["🗄️ SAP ERP System<br/>(Sales & Discount Data)"]
EXT["📥 Snowflake<br/>External Stage"]
RAW["raw.quotations<br/>raw.discount_conditions"]
STG["STAGING Layer<br/>stg_quotes<br/>stg_discount_conditions<br/>stg_product_master"]
TRF["TRANSFORM Layer<br/>trf_quotation<br/>trf_chatbot_quotation"]
DIM["Reference Dimensions<br/>dim_customer_discount_policy<br/>dim_product_portfolio<br/>fct_agreement_discounts"]
SEMA["SEMANTIC LAYER<br/>sv_quotation<br/>sv_chatbot_quotation"]
CORTEX["Cortex Analytics<br/>Guardrails & Information Protection<br/>Access Control<br/>Freshness Contract"]
UI["Snowflake UI<br/>(BI Dashboard)"]
CHAT["🤖 Internal Workspace<br/>(Chatbot)"]
SAP -->|Daily Batch 05:00 UTC| EXT
EXT -->|Load| RAW
RAW -->|Column mapping<br/>Type casting| STG
STG -->|Join dimensions<br/>Calculate discounts<br/>Apply governance flags| TRF
DIM -->|Enrich with policies| TRF
TRF -->|Expose semantic<br/>business logic<br/>Hide implementation| SEMA
SEMA -->|Apply guardrails<br/>Protect sensitive information<br/>Enforce freshness| CORTEX
CORTEX -->|Query & Visualize| UI
CORTEX -->|LLM consumption<br/>Prompt engineering layer| CHAT
CHAT -->|<1 min<br/>Quote Answers| USER["👤 Sales & Ops<br/>Team"]
UI -->|Dashboard<br/>Reporting| OPS["📊 Operations<br/>Team"]
SEMA -.->|Data Contract<br/>Lineage| AUDIT["🔒 Audit & Compliance<br/>- Access logs<br/>- Policy changes<br/>- Exception escalations"]
style SAP fill:#d4a574
style CORTEX fill:#ff9999
style SEMA fill:#99ccff
style CHAT fill:#99ff99
style USER fill:#ffcc99
style AUDIT fill:#cc99ff
classDef layer_raw fill:#e1e4e8
classDef layer_stg fill:#d0d7de
classDef layer_trf fill:#adb8c1
classDef layer_sema fill:#6e7681
class RAW layer_raw
class STG layer_stg
class TRF layer_trf
class SEMA layer_sema

View File

@@ -1,275 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Talk-to-Data: Architecture Animated Diagram</title>
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
<style>
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 20px;
background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
min-height: 100vh;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
border-radius: 8px;
box-shadow: 0 4px 6px rgba(0,0,0,0.1);
padding: 30px;
}
h1 {
color: #2c3e50;
text-align: center;
margin-bottom: 10px;
font-size: 28px;
}
.subtitle {
text-align: center;
color: #7f8c8d;
margin-bottom: 30px;
font-size: 14px;
}
.diagram-wrapper {
margin: 30px 0;
padding: 28px;
background: #f8f9fa;
border-radius: 6px;
border-left: 4px solid #3498db;
overflow-x: auto;
}
.mermaid {
display: flex;
justify-content: center;
min-width: 760px;
}
.mermaid svg {
width: 100%;
max-width: 920px;
height: auto;
overflow: hidden;
}
.mermaid .nodeLabel,
.mermaid .edgeLabel {
font-size: 16px !important;
}
.mermaid .edgeLabel {
background: transparent !important;
padding: 2px 4px;
}
.mermaid .edgeLabel:has(span:empty) {
display: none;
}
.legend {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 15px;
margin-top: 30px;
padding-top: 20px;
border-top: 1px solid #ecf0f1;
}
.legend-item {
display: flex;
align-items: center;
font-size: 13px;
}
.legend-color {
width: 20px;
height: 20px;
border-radius: 3px;
margin-right: 10px;
flex-shrink: 0;
}
.section {
margin-top: 40px;
padding: 20px;
background: #ecf0f1;
border-radius: 6px;
}
.section h2 {
color: #2c3e50;
margin-top: 0;
font-size: 18px;
border-bottom: 2px solid #3498db;
padding-bottom: 10px;
}
.section p {
color: #555;
line-height: 1.6;
margin: 10px 0;
}
.flow-step {
display: flex;
align-items: center;
margin: 15px 0;
padding: 10px;
background: white;
border-radius: 4px;
}
.flow-step span {
display: inline-block;
width: 110px;
min-width: 110px;
flex-shrink: 0;
font-weight: bold;
color: #3498db;
}
.flow-step p {
flex: 1;
min-width: 0;
}
code {
background: #f4f4f4;
padding: 2px 6px;
border-radius: 3px;
font-size: 12px;
}
@keyframes flow {
0% { opacity: 0.3; }
50% { opacity: 1; }
100% { opacity: 0.3; }
}
.animate-flow {
animation: flow 2s ease-in-out infinite;
}
</style>
</head>
<body>
<div class="container">
<h1>Talk-to-Data: Semantic Analytics Architecture</h1>
<p class="subtitle">SAP → Snowflake → dbt → Cortex → Chatbot | <1 minute quote answers</p>
<div class="diagram-wrapper">
<div class="mermaid">
graph TD
SAP["SAP ERP<br/>Quotes and discounts<br/>Daily batch"]
RAW["Snowflake raw data"]
STG["dbt staging<br/>Clean and standardize"]
TRF["dbt transforms<br/>Business rules and flags"]
SEMA["Semantic views<br/>Business terms and lineage"]
CORTEX["Cortex Analytics<br/>Guardrails and audit"]
UI["Snowflake UI<br/>Dashboards"]
CHAT["Internal workspace<br/>Chatbot"]
USER["Sales and operations<br/>30+ users<br/><1 min estimated answer"]
AUDIT["Audit trail<br/>Access and exceptions"]
SAP --> RAW
RAW --> STG
STG --> TRF
TRF --> SEMA
SEMA --> CORTEX
CORTEX --> UI
CORTEX --> CHAT
UI --> USER
CHAT --> USER
SEMA -.-> AUDIT
style SAP fill:#d4a574,stroke:#333,stroke-width:2px,color:#fff
style CORTEX fill:#ff9999,stroke:#c0392b,stroke-width:2px,color:#fff
style SEMA fill:#99ccff,stroke:#2980b9,stroke-width:2px,color:#fff
style CHAT fill:#99ff99,stroke:#27ae60,stroke-width:2px,color:#000
style USER fill:#ffcc99,stroke:#e67e22,stroke-width:2px,color:#000
style AUDIT fill:#cc99ff,stroke:#8e44ad,stroke-width:2px,color:#fff
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
style STG fill:#d0d7de,stroke:#666,stroke-width:1px
style TRF fill:#adb8c1,stroke:#666,stroke-width:1px
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
</div>
</div>
<div class="legend">
<div class="legend-item">
<div class="legend-color" style="background: #d4a574;"></div>
<span><strong>SAP Source:</strong> Daily batch extract</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #e1e4e8;"></div>
<span><strong>Raw:</strong> Unmodified data</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #d0d7de;"></div>
<span><strong>Staging:</strong> Data preparation</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #adb8c1;"></div>
<span><strong>Transform:</strong> Business logic</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #99ccff;"></div>
<span><strong>Semantic:</strong> Data contract</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #ff9999;"></div>
<span><strong>Cortex:</strong> AI guardrails</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #99ff99;"></div>
<span><strong>Chatbot:</strong> User interface</span>
</div>
<div class="legend-item">
<div class="legend-color" style="background: #cc99ff;"></div>
<span><strong>Audit:</strong> Compliance trail</span>
</div>
</div>
<div class="section">
<h2>Data Flow Stages</h2>
<div class="flow-step">
<span>1. Source</span>
<p>SAP ERP systems extract quotation, discount, and customer data daily at 05:00 UTC. Data includes quotation identifiers, line items, pricing, and discount conditions.</p>
</div>
<div class="flow-step">
<span>2. Raw</span>
<p>Snowflake External Stage ingests CSV/Parquet. Raw schema preserves source structure without modification. Data quality checks validate row counts, null patterns.</p>
</div>
<div class="flow-step">
<span>3. Staging</span>
<p><code>stg_*</code> models rename columns, cast data types, and handle nulls. Example: source quotation ID → <code>quote_id</code>, source amount → <code>net_value</code>. No business logic yet.</p>
</div>
<div class="flow-step">
<span>4. Transform</span>
<p><code>trf_quotation</code> joins customer policies, product portfolios, and agreement discounts. Calculates discount totals, applies governance flags (e.g., <code>discount_above_ceiling</code>). Two models: <code>trf_quotation</code> (BI) and <code>trf_chatbot_quotation</code> (LLM-optimized).</p>
</div>
<div class="flow-step">
<span>5. Semantic</span>
<p>Snowflake DDL semantic views expose curated dimensions and facts. Includes synonyms for natural language (e.g., "desconto cliente" → <code>discount_customer_pct</code>). Single source of truth for BI dashboards and LLM queries.</p>
</div>
<div class="flow-step">
<span>6. Cortex</span>
<p><a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst" target="_blank" rel="noopener noreferrer">Cortex Analyst</a> applies guardrails: hides raw discount percentages, protects customer names, checks data freshness, enforces role-based access, and logs all access. LLM queries go through here.</p>
</div>
<div class="flow-step">
<span>7. Consume</span>
<p><strong>Snowflake UI:</strong> Dashboards for ops/sales teams (full data visibility). <strong>Internal Workspace:</strong> Chatbot interface (guardrail-filtered data). Both consume the same semantic layer; governance rules ensure consistency.</p>
</div>
</div>
<div class="section">
<h2>Why This Architecture?</h2>
<p><strong>Governance-first:</strong> Guardrails (discount policies, protection of personal and customer information, and audit logs) are embedded in the data layer, not bolted on to the LLM prompt. The bot can't bypass policy; it's enforced by Cortex + dbt.</p>
<p><strong>Semantic contract:</strong> The semantic layer (sv_quotation, sv_chatbot_quotation) defines the agreement between data engineers, BI teams, and LLM consumers. Column definitions, synonyms, and access rules are version-controlled.</p>
<p><strong>Team efficiency:</strong> dbt lineage shows how every dimension flows from SAP → semantic layer. Data engineers and business analysts can trace a discount calculation back to SAP in seconds. Onboarding takes days, not weeks.</p>
<p><strong>Prototyping speed:</strong> Snowflake + Cortex means we didn't build custom infrastructure. POC shipped in weeks. Cortex guardrails were ready on day 1; no custom policy engine to build.</p>
</div>
<div class="section">
<h2>Expected Outcomes</h2>
<p><strong>Latency:</strong> Quote research from 5-30 minutes (email + manual lookup) → <1 minute (chatbot query)</p>
<p><strong>Scalability:</strong> 30+ salespeople can query the chatbot concurrently. Snowflake handles scale; dbt models are stateless (scale linearly).</p>
<p><strong>Governance:</strong> All discount exceptions flagged and audited. Policy violations are visible and traceable.</p>
<p><strong>Team capability:</strong> Once-daily batch + semantic layer pattern is reusable. Next use cases (contracts, pricing, serviceability) ship faster by leveraging the same playbook.</p>
</div>
<div class="section">
<h2>Learn More</h2>
<p>
See <code>README.md</code> for full narrative, decisions, and lessons learned.
Check <code>dbt/</code> for anonymized SQL models.
Review <code>cortex/cortex_governance_config.yaml</code> for guardrails and access control.
</p>
</div>
</div>
</body>
</html>

BIN
architecture_diagram.pdf Normal file

Binary file not shown.

Binary file not shown.

View File

@@ -22,20 +22,20 @@ cortex_policies:
# Access control: Who can query this semantic model # Access control: Who can query this semantic model
access_control: access_control:
default_role: "[REDACTED: SNOWFLAKE_ROLE]" default_role: "quotation_analytics_user"
authorized_teams: authorized_teams:
- name: "sales_team" - name: "sales_team"
snowflake_role: "[REDACTED: SALES_ROLE]" snowflake_role: "quotation_sales_user"
tables: ["sv_quotation", "sv_chatbot_quotation"] tables: ["sv_quotation", "sv_chatbot_quotation"]
max_rows_returned: 100000 max_rows_returned: 100000
- name: "operations_team" - name: "operations_team"
snowflake_role: "[REDACTED: OPS_ROLE]" snowflake_role: "quotation_operations_user"
tables: ["sv_quotation", "sv_chatbot_quotation"] tables: ["sv_quotation", "sv_chatbot_quotation"]
max_rows_returned: 500000 max_rows_returned: 500000
- name: "cortex_agent" - name: "cortex_agent"
snowflake_role: "[REDACTED: CORTEX_AGENT_ROLE]" snowflake_role: "quotation_cortex_agent"
tables: ["sv_chatbot_quotation"] tables: ["sv_chatbot_quotation"]
max_rows_returned: 1000 max_rows_returned: 1000
allowed_functions: ["semantic_search", "similarity_score"] allowed_functions: ["semantic_search", "similarity_score"]
@@ -59,7 +59,7 @@ cortex_policies:
description: "Mask actual customer names in chatbot responses" description: "Mask actual customer names in chatbot responses"
pattern: "customer_name|cliente_nome" pattern: "customer_name|cliente_nome"
action: "mask_value" action: "mask_value"
replacement: "[REDACTED: Customer Information]" replacement: "[Customer information protected]"
exception: "Sales team in authorized_teams can see unmasked values" exception: "Sales team in authorized_teams can see unmasked values"
- rule: "pii_scrubbing" - rule: "pii_scrubbing"
@@ -97,7 +97,7 @@ cortex_policies:
# Cost controls: Prevent runaway LLM usage # Cost controls: Prevent runaway LLM usage
cost_controls: cost_controls:
monthly_budget_usd: "[REDACTED: BUDGET]" monthly_budget_usd: "configured in the deployment environment"
alert_threshold_pct: 80 alert_threshold_pct: 80
per_query_max_tokens: 2000 per_query_max_tokens: 2000
max_concurrent_queries: 10 max_concurrent_queries: 10
@@ -121,9 +121,9 @@ semantic_model_lineage:
description: "How quotation data flows through transformations" description: "How quotation data flows through transformations"
stages: stages:
1_source: 1_source:
system: "[REDACTED: SAP_SYSTEM]" system: "enterprise_erp_source"
frequency: "Daily batch 05:00 UTC" frequency: "Daily batch 05:00 UTC"
tables: ["[REDACTED: SOURCE_QUOTE_TABLE]", "[REDACTED: SOURCE_DISCOUNT_TABLE]"] tables: ["source_quotes", "source_discount_conditions"]
2_staging: 2_staging:
schema: "STAGING" schema: "STAGING"
@@ -142,8 +142,8 @@ semantic_model_lineage:
5_cortex_consumption: 5_cortex_consumption:
interface_1: "Snowflake Native App (BI)" interface_1: "Snowflake Native App (BI)"
interface_2: "[REDACTED: SIEMENS_WORKSPACE]" interface_2: "internal_chatbot_workspace"
llm_model: "[REDACTED: LLM_VERSION]" llm_model: "managed_cortex_model"
prompt_template: | prompt_template: |
You are a sales support assistant. Answer quote questions using only You are a sales support assistant. Answer quote questions using only
the data provided. If discount exceeds policy, flag for human review. the data provided. If discount exceeds policy, flag for human review.

View File

@@ -1,43 +0,0 @@
# LinkedIn post draft
Sales teams should not need an email chain to answer a routine quotation question.
In a proof of concept, I designed a governance-first semantic analytics flow for quotation data:
**SAP → Snowflake → dbt → Semantic Views → Cortex → Chatbot**
The business problem was simple: more than 30 salespeople could need help validating a quote, discount, or product classification. A question that looked small could take 5–30 minutes because the answer depended on manual lookup and internal communication.
The solution was not “put an LLM on top of raw tables.”
I first created a semantic contract:
- curated quotation dimensions and facts;
- business-friendly synonyms for natural-language questions;
- centralized discount calculations;
- explicit policy-exception flags;
- lineage from source data to the consumer-facing model.
Then Cortex applied the controls around that contract:
- data freshness warning for the daily batch;
- role-based access;
- protection of sensitive information;
- escalation when a discount exceeded policy;
- audit logging for queries and decisions.
The semantic model was consumed in two ways:
1. Snowflake's interface for analytical exploration.
2. An internal chatbot workspace for direct questions, with prompt engineering to refine the response experience.
The estimated user outcome was reducing routine quote research from 5–30 minutes to less than one minute. That is an estimate based on the existing human workflow, not a production benchmark—and that distinction matters.
The main lesson: **governance belongs in the data and semantic layers, not only in the prompt.**
The POC also created a reusable delivery pattern for future use cases: define the business contract, expose only the right data, add guardrails, then choose the lightest useful interface.
I documented the anonymized architecture, decisions, and lessons learned here:
[GitHub repository link]
#DataArchitecture #Snowflake #dbt #DataGovernance #EnterpriseAI #TechnicalDelivery

View File

@@ -9,17 +9,17 @@
with source_quotes as ( with source_quotes as (
select select
[REDACTED: quote_id column], quote_id,
[REDACTED: line_item column], line_item,
[REDACTED: customer_id column], customer_id,
[REDACTED: material_id column], material_id,
[REDACTED: quantity column], quantity,
[REDACTED: net_value column], net_value,
[REDACTED: quote_date column], quote_date,
[REDACTED: sales_person_id column], sales_person_id,
[REDACTED: source_timestamp column] source_timestamp
from [REDACTED: SOURCE_SCHEMA].[REDACTED: QUOTE_TABLE] from source_quotes
where [REDACTED: date_partition] >= dateadd(day, -1, current_date()) where quote_date >= dateadd(day, -1, current_date())
), ),
enrich_product_portfolio as ( enrich_product_portfolio as (
@@ -34,9 +34,9 @@ enrich_product_portfolio as (
sq.sales_person_id, sq.sales_person_id,
/* Product portfolio classification */ /* Product portfolio classification */
case case
when sq.material_id like '[REDACTED: pattern 1]%' then 'data-integration' when sq.material_id like 'DI-%' then 'data-integration'
when sq.material_id like '[REDACTED: pattern 2]%' then 'automation' when sq.material_id like 'AU-%' then 'automation'
when sq.material_id like '[REDACTED: pattern 3]%' then 'connectivity' when sq.material_id like 'CO-%' then 'connectivity'
else 'other' else 'other'
end as portfolio_code, end as portfolio_code,
pm.portfolio_name, pm.portfolio_name,
@@ -109,5 +109,3 @@ select
sales_person_id, sales_person_id,
current_timestamp() as dbt_loaded_at current_timestamp() as dbt_loaded_at
from governance_flags from governance_flags
-- ponytail: materialized as table for chatbot + BI queries (not incremental yet; append-only refresh if needed later)