gabspereira/talk-to-data-case-study
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

review docs

Browse Source
This commit is contained in:
@gabriel.pereira
2026-09-11 11:18:14 -03:00
parent cf194dc908
commit f4dcd73676
3 changed files with 21 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@@ -3,4 +3,3 @@ case_study.css
case_study_one_page.html
case_study_one_page.md
architecture_dag.md
linkedin_post.md

73
README.md
View File

@@ -1,6 +1,6 @@
# Talk-to-Data: Semantic Analytics for Enterprise Quotations
**A POC case study in governance-first semantic layer design: SAP → Snowflake → Cortex → Chatbot.**
**A PoC case study in governance-first semantic layer design: SAP → Snowflake → Cortex → Chatbot.**
---
@@ -25,44 +25,44 @@ Sales and operations teams across multiple regions handle thousands of quote req
```
┌─────────────────────────────────────────────────────────────┐
│ SAP ERP (Sales, Pricing, Discounts) │
│ Updated daily via batch extract (05:00 UTC) │
│ Updated daily via batch extract (05:00 UTC) │
└────────────────────────┬────────────────────────────────────┘
│
Daily ELT Job
│
┌────────────────────────▼────────────────────────────────────┐
│ Snowflake Raw Layer │
│ - raw.quotations (document, line item, amount, ...) │
│ - raw.discount_conditions (discount tiers, policies) │
│ - raw.customer_master (customer codes, agreements) │
│ - raw.quotations (document, line item, amount, ...) │
│ - raw.discount_conditions (discount tiers, policies) │
│ - raw.customer_master (customer codes, agreements) │
└────────────────────────┬────────────────────────────────────┘
│
dbt Staging
│
┌────────────────────────▼────────────────────────────────────┐
│ Staging Layer (STAGING schema) │
│ - Column mapping, type casting, null handling │
│ - No business logic, just data preparation │
│ - Column mapping, type casting, null handling │
│ - No business logic, just data preparation │
└────────────────────────┬────────────────────────────────────┘
│
dbt Transforms
│
┌────────────────────────▼────────────────────────────────────┐
│ Transform Layer (TRANSFORM schema) │
│ - Join discount conditions, policies, agreements │
│ - Calculate: discount tiers, totals, exceptions │
│ - Apply: governance flags, audit columns │
│ - Models: trf_quotation, trf_chatbot_quotation │
│ - Join discount conditions, policies, agreements │
│ - Calculate: discount tiers, totals, exceptions │
│ - Apply: governance flags, audit columns │
│ - Models: trf_quotation, trf_chatbot_quotation │
└────────────────────────┬────────────────────────────────────┘
│
Semantic Views (DDL)
│
┌────────────────────────▼────────────────────────────────────┐
│ Semantic Layer (DISTRIBUTE_DDL schema) │
│ - sv_quotation: Full BI consumption │
│ - sv_chatbot_quotation: LLM-optimized dimensions │
│ - Exposes: business logic (discounts, policies) │
│ - Hides: implementation details (temp tables, keys) │
│ - sv_quotation: Full BI consumption │
│ - sv_chatbot_quotation: LLM-optimized dimensions │
│ - Exposes: business logic (discounts, policies) │
│ - Hides: implementation details (temp tables, keys) │
└────────────────────────┬────────────────────────────────────┘
│
Cortex Analytics Guardrails
@@ -76,8 +76,8 @@ Sales and operations teams across multiple regions handle thousands of quote req
│ - Visual analysis │ │ - Quote Q&A │
│ - Sales reporting │ │ - Prompt engineering │
│ - Ops monitoring │ │ - Real-time answers │
└───────────┬──────────┘ └──────────┬────────────┘
│ │
└───────────┬───── ─────┘ └──────────┬────────────┘
│ │
│ <1 minute turnaround
│
End Users (Sales, Ops)
@@ -178,7 +178,7 @@ Governance is not a checklist; it's embedded in the data layer.
---
## Results: From POC to Impact
## Results: From PoC to Impact
### Estimated Impact (Based on User Interviews)
- **Quote turnaround**: 5-30 min (email + manual lookup) → <1 min (chatbot query)
@@ -186,7 +186,7 @@ Governance is not a checklist; it's embedded in the data layer.
- Pilots with 5 sales reps: ~50+ hours/week freed (estimated)
- Extrapolation: 30+ sales team × 5 hours/week = 150+ hours recovered org-wide
- **Error reduction**: Discount exceptions caught by policy guardrails (vs. manually reviewed)
- **Team velocity**: POC shipped in weeks (not months), governance patterns established for scale
- **Team velocity**: PoC shipped in weeks (not months), governance patterns established for scale
### Why This Matters for Delivery Managers
1. **Governance-first approach**: We didn't build a prototype and hope for policy later. Guardrails were baked in from day one.
@@ -231,10 +231,6 @@ Governance is not a checklist; it's embedded in the data layer.
✅ **dbt for team alignment**: SQL-first + schema docs meant the data team and business stakeholders spoke the same language. Onboarding new people was fast.
### What We'd Change at Scale
⚠️ **Real-time discount updates**: If pilot shows sales need <1hr freshness on policy changes, escalate to hourly ELT.
⚠️ **Multi-region governance**: Current config is single-region (BR). If expanding to US/CA, add region-specific discount tiers and approval workflows.
⚠️ **Cortex cost**: Monitor Cortex token usage as chatbot volume scales. Current config has cost controls (monthly budget, per-query max tokens); may need aggressive pruning at scale.
⚠️ **Semantic view complexity**: If adding more dimensions (15→50), consider splitting into focused semantic views (e.g., `sv_quote_discounts`, `sv_quote_compliance`) to keep queries fast.
@@ -263,37 +259,10 @@ A: Yes. Adapt the dbt dialect (BigQuery: `jinja-sql`, Redshift: `redshift` profi
**Q: Is once-daily refresh really enough?**
A: For quote research, yes. If you need real-time pricing updates, escalate to hourly ELT. The architecture supports it; just change the schedule in your orchestrator.
**Q: How do I handle discount exceptions?**
A: Cortex policy flags them and escalates to a human review queue (max 10/hour). Ops team approves or rejects in Snowflake; bot learns the decision for future similar quotes.
**Q: What if I need to scale to 500+ users?**
A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same.
A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same. The real issue comes when you need RLS requirements, this will enforce you to manager with additional tools or methods beyond Snowflake RBAC (Role Based Access Control).
**Q: Can I export this as a Snowflake Native App for partners?**
A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data.
A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data. Cortex Analyst can be seamlessly integrated into any application using REST API.
---
## Next Steps
1. **Expand pilot**: Rollout to 30+ sales team over 4 weeks
2. **Measure impact**: Track quote turnaround times, bot usage, escalation rates
3. **Iterate on policies**: Refine discount rules and information-protection controls based on pilot feedback
4. **Add new domains**: Reuse the semantic layer pattern for contracts, serviceability, pricing
5. **Scale LLM**: Move from proof-of-concept to production volume (monitor cost, latency)
---
## Contact & Questions
This case study demonstrates:
- Governance-first architecture for enterprise AI
- Semantic layer design (SAP → dbt → Snowflake → LLM)
- Operationalizing guardrails and audit trails
- Shipping production-ready POCs in weeks
For questions about the design decisions, data flow, or how to adapt this to your domain, reach out.
---
**Co-authored by Copilot**

43
linkedin_post.md
View File

@@ -1,43 +0,0 @@
# LinkedIn post draft
Sales teams should not need an email chain to answer a routine quotation question.
In a proof of concept, I designed a governance-first semantic analytics flow for quotation data:
**SAP → Snowflake → dbt → Semantic Views → Cortex → Chatbot**
The business problem was simple: more than 30 salespeople could need help validating a quote, discount, or product classification. A question that looked small could take 5–30 minutes because the answer depended on manual lookup and internal communication.
The solution was not “put an LLM on top of raw tables.”
I first created a semantic contract:
- curated quotation dimensions and facts;
- business-friendly synonyms for natural-language questions;
- centralized discount calculations;
- explicit policy-exception flags;
- lineage from source data to the consumer-facing model.
Then Cortex applied the controls around that contract:
- data freshness warning for the daily batch;
- role-based access;
- protection of sensitive information;
- escalation when a discount exceeded policy;
- audit logging for queries and decisions.
The semantic model was consumed in two ways:
1. Snowflake's interface for analytical exploration.
2. An internal chatbot workspace for direct questions, with prompt engineering to refine the response experience.
The estimated user outcome was reducing routine quote research from 5–30 minutes to less than one minute. That is an estimate based on the existing human workflow, not a production benchmark—and that distinction matters.
The main lesson: **governance belongs in the data and semantic layers, not only in the prompt.**
The POC also created a reusable delivery pattern for future use cases: define the business contract, expose only the right data, add guardrails, then choose the lightest useful interface.
I documented the anonymized architecture, decisions, and lessons learned here:
[GitHub repository link]
#DataArchitecture #Snowflake #dbt #DataGovernance #EnterpriseAI #TechnicalDelivery