docs: add talk-to-data case study
Add anonymized architecture, governance examples, diagrams, and interview materials.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
1
.gitignore
vendored
Normal file
1
.gitignore
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
local/
|
||||||
299
README.md
Normal file
299
README.md
Normal file
@@ -0,0 +1,299 @@
|
|||||||
|
# Talk-to-Data: Semantic Analytics for Enterprise Quotations
|
||||||
|
|
||||||
|
**A POC case study in governance-first semantic layer design: SAP → Snowflake → Cortex → Chatbot.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
Sales and operations teams across multiple regions handle thousands of quote requests monthly. Each inquiry requires:
|
||||||
|
- **Manual research**: Discount policy lookups, product classification, customer contract data
|
||||||
|
- **Cross-system queries**: SAP for pricing, internal databases for agreements, email chains with ops
|
||||||
|
- **Coordination overhead**: Sales emails ops, ops researches and replies (5-30 minutes per quote)
|
||||||
|
- **Scale challenge**: 30+ salespersons generating inquiries daily across the sales organization
|
||||||
|
|
||||||
|
**Result**: Quote turnaround times of 5-30 minutes, even for routine requests. Ops team context-switches between email, SAP, and spreadsheets. Sales can't get answers fast enough to close deals.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Solution: Semantic Analytics Layer + Cortex Guardrails
|
||||||
|
|
||||||
|
**Core insight**: Build a semantic layer that serves as the single source of truth for quote logic, then consume it via two interfaces: BI dashboards and an LLM-powered chatbot. Governance lives in the data layer, not in prompts.
|
||||||
|
|
||||||
|
### Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ SAP ERP (Sales, Pricing, Discounts) │
|
||||||
|
│ Updated daily via batch extract (05:00 UTC) │
|
||||||
|
└────────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
Daily ELT Job
|
||||||
|
│
|
||||||
|
┌────────────────────────▼────────────────────────────────────┐
|
||||||
|
│ Snowflake Raw Layer │
|
||||||
|
│ - raw.quotations (document, line item, amount, ...) │
|
||||||
|
│ - raw.discount_conditions (discount tiers, policies) │
|
||||||
|
│ - raw.customer_master (customer codes, agreements) │
|
||||||
|
└────────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
dbt Staging
|
||||||
|
│
|
||||||
|
┌────────────────────────▼────────────────────────────────────┐
|
||||||
|
│ Staging Layer (STAGING schema) │
|
||||||
|
│ - Column mapping, type casting, null handling │
|
||||||
|
│ - No business logic, just data preparation │
|
||||||
|
└────────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
dbt Transforms
|
||||||
|
│
|
||||||
|
┌────────────────────────▼────────────────────────────────────┐
|
||||||
|
│ Transform Layer (TRANSFORM schema) │
|
||||||
|
│ - Join discount conditions, policies, agreements │
|
||||||
|
│ - Calculate: discount tiers, totals, exceptions │
|
||||||
|
│ - Apply: governance flags, audit columns │
|
||||||
|
│ - Models: trf_quotation, trf_chatbot_quotation │
|
||||||
|
└────────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
Semantic Views (DDL)
|
||||||
|
│
|
||||||
|
┌────────────────────────▼────────────────────────────────────┐
|
||||||
|
│ Semantic Layer (DISTRIBUTE_DDL schema) │
|
||||||
|
│ - sv_quotation: Full BI consumption │
|
||||||
|
│ - sv_chatbot_quotation: LLM-optimized dimensions │
|
||||||
|
│ - Exposes: business logic (discounts, policies) │
|
||||||
|
│ - Hides: implementation details (temp tables, keys) │
|
||||||
|
└────────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
Cortex Analytics Guardrails
|
||||||
|
│
|
||||||
|
┌───────────────┴───────────────┐
|
||||||
|
│ │
|
||||||
|
┌────────▼──────────────┐ ┌─────────▼─────────────┐
|
||||||
|
│ Snowflake UI │ │ Internal Workspace │
|
||||||
|
│ (BI Dashboard) │ │ (Chatbot + LLM) │
|
||||||
|
│ │ │ │
|
||||||
|
│ - Visual analysis │ │ - Quote Q&A │
|
||||||
|
│ - Sales reporting │ │ - Prompt engineering │
|
||||||
|
│ - Ops monitoring │ │ - Real-time answers │
|
||||||
|
└───────────┬──────────┘ └──────────┬────────────┘
|
||||||
|
│ │
|
||||||
|
│ <1 minute turnaround
|
||||||
|
│
|
||||||
|
End Users (Sales, Ops)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stack: Why Each Layer?
|
||||||
|
|
||||||
|
### SAP → Snowflake ELT (Daily Batch)
|
||||||
|
- **Why**: Quotation data lives in SAP; Snowflake is the reporting warehouse
|
||||||
|
- **How**: Daily batch extract (05:00 UTC), once-per-day refresh sufficient for quote workflow (real-time inventory not required)
|
||||||
|
- **Trade-off**: Once-daily freshness OK for most quote requests; could escalate to hourly for real-time dashboard if needed
|
||||||
|
|
||||||
|
### Staging → Transform → Semantic Layer (dbt)
|
||||||
|
- **Why dbt?**
|
||||||
|
- **Lineage**: Column-level traceability (SAP field → transform → semantic dimension)
|
||||||
|
- **Testability**: Unit tests on discount calculations, data quality checks
|
||||||
|
- **Team reuse**: SQL-first approach, skill portability across data team
|
||||||
|
- **Governance**: Schema.yml documents business logic and access rules
|
||||||
|
- **Reproducibility**: Version-controlled transformations, audit trail of changes
|
||||||
|
|
||||||
|
- **Why separate layers?**
|
||||||
|
- **Staging**: Decouples source-system brittleness from business logic
|
||||||
|
- **Transform**: Concentrates business rules (discount math, policy flags) in one place
|
||||||
|
- **Semantic**: Exposes curated dimensions to downstream consumers; hides technical debt
|
||||||
|
|
||||||
|
### Snowflake Semantic Views
|
||||||
|
- **What**: Semantic views (DDL) wrap transform models; define dimensions, facts, and synonyms
|
||||||
|
- **Why**:
|
||||||
|
- Define the contract between data and consumers (Sales, Cortex)
|
||||||
|
- Synonyms enable natural-language queries ("desconto cliente" → `discount_customer_pct`)
|
||||||
|
- Consistent schema across BI dashboards and LLM consumption
|
||||||
|
- **Example**: A chatbot user asks "What's the customer discount on this quote?" The semantic layer maps "customer discount" (natural language) to `discount_customer_pct` (column), and Cortex applies guardrails (hides the raw percentage and shows only "within policy" or "requires approval")
|
||||||
|
|
||||||
|
### Cortex Analytics + Guardrails
|
||||||
|
- **What**: Snowflake's native LLM platform + custom governance policies
|
||||||
|
- **Why Cortex, not custom LLM?**
|
||||||
|
- Built-in guardrails: protection of sensitive information, access control, and cost controls
|
||||||
|
- Fast prototyping: Cortex agents consume Snowflake semantic views directly (no ETL to another system)
|
||||||
|
- Governance-as-code: Policies live alongside data, versioned, audited
|
||||||
|
- Security: Data stays in Snowflake; LLM calls don't leak sensitive data
|
||||||
|
- **Trade-off**: Cortex is Snowflake-specific (vs. generic LLM + Anthropic); acceptable for POC, revisit if multi-cloud needed later
|
||||||
|
|
||||||
|
### Dual Consumption (Snowflake UI + Internal Workspace)
|
||||||
|
- **Snowflake UI**: Native BI dashboards for sales/ops teams. Semantic model feeds Power BI or Tableau directly.
|
||||||
|
- **Internal Workspace**: Existing chatbot environment. Cortex queries the semantic layer + prompt engineering layer refines responses.
|
||||||
|
- **Why two interfaces?** Users consume data differently — dashboards for exploration, chatbot for direct answers. Semantic layer unified the source, reducing maintenance.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Decisions & Trade-offs
|
||||||
|
|
||||||
|
| Decision | Rationale | Trade-off |
|
||||||
|
|----------|-----------|----------|
|
||||||
|
| **Governance-first** | Discount policies, protection of personal and customer information, and exception escalation built into the data layer | Requires upfront effort to define policies; can't skip it for "speed" |
|
||||||
|
| **Semantic views (DDL)** | Single contract between data → BI/LLM | Requires mapping business terms to columns; initial work to get synonyms right |
|
||||||
|
| **Once-daily batch** | Sufficient for quote workflow; reduces ELT complexity | Not real-time; acceptable for POC; upgrade to hourly if pilot shows need |
|
||||||
|
| **Cortex (vs. custom LLM)** | Fast prototyping, built-in guardrails, stays in Snowflake ecosystem | Cortex-only (could integrate with Anthropic later) |
|
||||||
|
| **Separate transform model per interface** | `trf_quotation` for BI, `trf_chatbot_quotation` for LLM — optimize each | Slight duplication; worth it if interfaces have different freshness/cardinality needs |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Governance: How We Built Safety In
|
||||||
|
|
||||||
|
Governance is not a checklist; it's embedded in the data layer.
|
||||||
|
|
||||||
|
### 1. **Discount Policy Guardrails**
|
||||||
|
- **Rule**: No discount exceeds the configured policy ceiling
|
||||||
|
- **Implementation**:
|
||||||
|
- Transform model calculates `discount_above_ceiling` flag
|
||||||
|
- Cortex policy hides raw discount percentages; shows only "Within Policy" or "Requires Approval"
|
||||||
|
- Chatbot escalates flagged quotes to ops team (max 10 escalations/hour to prevent bot spam)
|
||||||
|
- **Effect**: Sales can't accidentally propose an unapproved discount; exceptions are visible and auditable
|
||||||
|
|
||||||
|
### 2. **Protection of Personal and Customer Information**
|
||||||
|
- **Rule**: The chatbot must not expose customer names, contact details, or negotiated terms unless the user is authorized
|
||||||
|
- **Implementation**:
|
||||||
|
- `sv_chatbot_quotation` includes only aggregated customer data (customer_id, portfolio_code)
|
||||||
|
- Cortex policy replaces `customer_name` with "[Customer information protected]" in chatbot responses
|
||||||
|
- Exception: Sales team in authorized role can see unmasked values via dashboard
|
||||||
|
- **Effect**: Chatbot can answer quote questions without leaking customer contracts
|
||||||
|
|
||||||
|
### 3. **Access Control & Audit Trail**
|
||||||
|
- **Rule**: Different roles see different data
|
||||||
|
- Sales: Can query `sv_chatbot_quotation` (limited fields, no raw discounts)
|
||||||
|
- Ops: Can query `sv_quotation` (full discount details, for policy review)
|
||||||
|
- Cortex Agent: Can query `sv_chatbot_quotation` only (locked down for LLM)
|
||||||
|
- **Implementation**: Snowflake role-based access control (RBAC) + Cortex `authorized_teams` config
|
||||||
|
- **Audit**: All queries logged; policy violations trigger alerts
|
||||||
|
|
||||||
|
### 4. **Data Freshness Contract**
|
||||||
|
- **Rule**: Chatbot rejects queries on stale data; warns user if data is > 24 hours old
|
||||||
|
- **Implementation**:
|
||||||
|
- dbt model adds `load_timestamp` column
|
||||||
|
- Cortex quality gate checks: `current_timestamp() - load_timestamp < 24 hours`
|
||||||
|
- Failure: Returns "Data unavailable; please retry in 1 hour" (prevents stale-data surprises)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Results: From POC to Impact
|
||||||
|
|
||||||
|
### Estimated Impact (Based on User Interviews)
|
||||||
|
- **Quote turnaround**: 5-30 min (email + manual lookup) → <1 min (chatbot query)
|
||||||
|
- Savings per quote: 5-30 minutes
|
||||||
|
- Pilots with 5 sales reps: ~50+ hours/week freed (estimated)
|
||||||
|
- Extrapolation: 30+ sales team × 5 hours/week = 150+ hours recovered org-wide
|
||||||
|
- **Error reduction**: Discount exceptions caught by policy guardrails (vs. manually reviewed)
|
||||||
|
- **Team velocity**: POC shipped in weeks (not months), governance patterns established for scale
|
||||||
|
|
||||||
|
### Why This Matters for Delivery Managers
|
||||||
|
1. **Governance-first approach**: We didn't build a prototype and hope for policy later. Guardrails were baked in from day one.
|
||||||
|
2. **Reusable patterns**: The semantic layer + Cortex + dbt playbook works for any quote/contract/discount domain. Next use cases ship faster.
|
||||||
|
3. **Measurable before scale**: We validated impact with a small pilot; ready to scale to 30+ users with confidence.
|
||||||
|
4. **Ops visibility**: Sales/ops can see what the bot is doing (audit logs, escalation flags); not a black box.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How to Use This Repo
|
||||||
|
|
||||||
|
### Deliverables
|
||||||
|
- **`dbt/`**: Anonymized dbt models (staging, transforms, semantic views)
|
||||||
|
- `models/staging/stg_*.sql` — Raw layer transformations
|
||||||
|
- `models/transform/trf_*.sql` — Business logic
|
||||||
|
- `models/distribute_ddl/sv_*.yml` — Semantic views with synonyms
|
||||||
|
- **`cortex/`**: Governance config (information protection, access control, audit rules)
|
||||||
|
- `cortex_governance_config.yaml` — Guardrails, freshness contracts, LLM policies
|
||||||
|
- **`docs/`**: Architecture diagrams, decision logs
|
||||||
|
- `architecture_dag.md` — Data flow Mermaid diagram
|
||||||
|
- `decisions.md` — Architecture decision records (ADRs)
|
||||||
|
- **`README.md`**: This file (the narrative for interviewers/stakeholders)
|
||||||
|
|
||||||
|
### How to Adapt This to Your Domain
|
||||||
|
1. **Replace discount logic** with your domain (e.g., pricing tiers, contract terms, approval workflows)
|
||||||
|
2. **Adjust freshness** (once-daily to hourly/real-time) based on your use case
|
||||||
|
3. **Modify semantic dimensions** to match your business terminology
|
||||||
|
4. **Update Cortex policies** for your personal-information and governance rules
|
||||||
|
5. **Test with a small pilot** (5-10 users) before org-wide rollout
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Lessons Learned
|
||||||
|
|
||||||
|
### What Worked
|
||||||
|
✅ **Semantic views as the contract**: Defining dimensions + synonyms upfront saved rework later. Sales/ops understood the model; Cortex had clear field mappings.
|
||||||
|
|
||||||
|
✅ **Governance in the data layer**: Embedding discount flags and information-protection rules in dbt/Cortex meant the bot couldn't bypass them. No workaround hacks possible.
|
||||||
|
|
||||||
|
✅ **Once-daily batch sufficient**: For this workflow, 24-hour freshness was OK. We didn't need real-time complexity (yet).
|
||||||
|
|
||||||
|
✅ **dbt for team alignment**: SQL-first + schema docs meant the data team and business stakeholders spoke the same language. Onboarding new people was fast.
|
||||||
|
|
||||||
|
### What We'd Change at Scale
|
||||||
|
⚠️ **Real-time discount updates**: If pilot shows sales need <1hr freshness on policy changes, escalate to hourly ELT.
|
||||||
|
|
||||||
|
⚠️ **Multi-region governance**: Current config is single-region (BR). If expanding to US/CA, add region-specific discount tiers and approval workflows.
|
||||||
|
|
||||||
|
⚠️ **Cortex cost**: Monitor Cortex token usage as chatbot volume scales. Current config has cost controls (monthly budget, per-query max tokens); may need aggressive pruning at scale.
|
||||||
|
|
||||||
|
⚠️ **Semantic view complexity**: If adding more dimensions (15→50), consider splitting into focused semantic views (e.g., `sv_quote_discounts`, `sv_quote_compliance`) to keep queries fast.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tech Stack Summary
|
||||||
|
|
||||||
|
| Component | Tool | Why |
|
||||||
|
|-----------|------|-----|
|
||||||
|
| **Data Warehouse** | Snowflake | Scalable, built-in Cortex, governance features |
|
||||||
|
| **Transformations** | dbt + Jinja | SQL lineage, testability, team skill reuse |
|
||||||
|
| **Semantic Layer** | Snowflake DDL | Data contract, synonyms for NLP, single source of truth |
|
||||||
|
| **AI/Guardrails** | Cortex Analytics | Native LLM, information protection, access control, cost controls |
|
||||||
|
| **Consumption 1** | Snowflake Native App / BI Tool | Dashboard for ops/sales exploration |
|
||||||
|
| **Consumption 2** | Internal Workspace + Prompt Engineering | Chatbot for direct question-answering |
|
||||||
|
| **Infrastructure** | Cloud-native Snowflake | No ops overhead, pay-per-query |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ
|
||||||
|
|
||||||
|
**Q: Can I use this with a different warehouse (e.g., BigQuery, Redshift)?**
|
||||||
|
A: Yes. Adapt the dbt dialect (BigQuery: `jinja-sql`, Redshift: `redshift` profile). Cortex is Snowflake-native; substitute with your own LLM guardrails (policy enforcement layer).
|
||||||
|
|
||||||
|
**Q: Is once-daily refresh really enough?**
|
||||||
|
A: For quote research, yes. If you need real-time pricing updates, escalate to hourly ELT. The architecture supports it; just change the schedule in your orchestrator.
|
||||||
|
|
||||||
|
**Q: How do I handle discount exceptions?**
|
||||||
|
A: Cortex policy flags them and escalates to a human review queue (max 10/hour). Ops team approves or rejects in Snowflake; bot learns the decision for future similar quotes.
|
||||||
|
|
||||||
|
**Q: What if I need to scale to 500+ users?**
|
||||||
|
A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same.
|
||||||
|
|
||||||
|
**Q: Can I export this as a Snowflake Native App for partners?**
|
||||||
|
A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. **Expand pilot**: Rollout to 30+ sales team over 4 weeks
|
||||||
|
2. **Measure impact**: Track quote turnaround times, bot usage, escalation rates
|
||||||
|
3. **Iterate on policies**: Refine discount rules and information-protection controls based on pilot feedback
|
||||||
|
4. **Add new domains**: Reuse the semantic layer pattern for contracts, serviceability, pricing
|
||||||
|
5. **Scale LLM**: Move from proof-of-concept to production volume (monitor cost, latency)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contact & Questions
|
||||||
|
|
||||||
|
This case study demonstrates:
|
||||||
|
- Governance-first architecture for enterprise AI
|
||||||
|
- Semantic layer design (SAP → dbt → Snowflake → LLM)
|
||||||
|
- Operationalizing guardrails and audit trails
|
||||||
|
- Shipping production-ready POCs in weeks
|
||||||
|
|
||||||
|
For questions about the design decisions, data flow, or how to adapt this to your domain, reach out.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Co-authored by Copilot**
|
||||||
59
architecture_dag.md
Normal file
59
architecture_dag.md
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
# Quotation Semantic Layer - Data Flow DAG
|
||||||
|
# SAP Source -> Snowflake -> dbt Transforms -> Semantic Views -> Cortex -> Chatbot Output
|
||||||
|
|
||||||
|
graph LR
|
||||||
|
SAP["🗄️ SAP ERP System<br/>(Sales & Discount Data)"]
|
||||||
|
|
||||||
|
EXT["📥 Snowflake<br/>External Stage"]
|
||||||
|
|
||||||
|
RAW["raw.quotations<br/>raw.discount_conditions"]
|
||||||
|
|
||||||
|
STG["STAGING Layer<br/>stg_quotes<br/>stg_discount_conditions<br/>stg_product_master"]
|
||||||
|
|
||||||
|
TRF["TRANSFORM Layer<br/>trf_quotation<br/>trf_chatbot_quotation"]
|
||||||
|
|
||||||
|
DIM["Reference Dimensions<br/>dim_customer_discount_policy<br/>dim_product_portfolio<br/>fct_agreement_discounts"]
|
||||||
|
|
||||||
|
SEMA["SEMANTIC LAYER<br/>sv_quotation<br/>sv_chatbot_quotation"]
|
||||||
|
|
||||||
|
CORTEX["Cortex Analytics<br/>Guardrails & Information Protection<br/>Access Control<br/>Freshness Contract"]
|
||||||
|
|
||||||
|
UI["Snowflake UI<br/>(BI Dashboard)"]
|
||||||
|
CHAT["🤖 Internal Workspace<br/>(Chatbot)"]
|
||||||
|
|
||||||
|
SAP -->|Daily Batch 05:00 UTC| EXT
|
||||||
|
EXT -->|Load| RAW
|
||||||
|
|
||||||
|
RAW -->|Column mapping<br/>Type casting| STG
|
||||||
|
|
||||||
|
STG -->|Join dimensions<br/>Calculate discounts<br/>Apply governance flags| TRF
|
||||||
|
DIM -->|Enrich with policies| TRF
|
||||||
|
|
||||||
|
TRF -->|Expose semantic<br/>business logic<br/>Hide implementation| SEMA
|
||||||
|
|
||||||
|
SEMA -->|Apply guardrails<br/>Protect sensitive information<br/>Enforce freshness| CORTEX
|
||||||
|
|
||||||
|
CORTEX -->|Query & Visualize| UI
|
||||||
|
CORTEX -->|LLM consumption<br/>Prompt engineering layer| CHAT
|
||||||
|
|
||||||
|
CHAT -->|<1 min<br/>Quote Answers| USER["👤 Sales & Ops<br/>Team"]
|
||||||
|
UI -->|Dashboard<br/>Reporting| OPS["📊 Operations<br/>Team"]
|
||||||
|
|
||||||
|
SEMA -.->|Data Contract<br/>Lineage| AUDIT["🔒 Audit & Compliance<br/>- Access logs<br/>- Policy changes<br/>- Exception escalations"]
|
||||||
|
|
||||||
|
style SAP fill:#d4a574
|
||||||
|
style CORTEX fill:#ff9999
|
||||||
|
style SEMA fill:#99ccff
|
||||||
|
style CHAT fill:#99ff99
|
||||||
|
style USER fill:#ffcc99
|
||||||
|
style AUDIT fill:#cc99ff
|
||||||
|
|
||||||
|
classDef layer_raw fill:#e1e4e8
|
||||||
|
classDef layer_stg fill:#d0d7de
|
||||||
|
classDef layer_trf fill:#adb8c1
|
||||||
|
classDef layer_sema fill:#6e7681
|
||||||
|
|
||||||
|
class RAW layer_raw
|
||||||
|
class STG layer_stg
|
||||||
|
class TRF layer_trf
|
||||||
|
class SEMA layer_sema
|
||||||
275
architecture_diagram.html
Normal file
275
architecture_diagram.html
Normal file
@@ -0,0 +1,275 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<title>Talk-to-Data: Architecture Animated Diagram</title>
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
|
||||||
|
<style>
|
||||||
|
body {
|
||||||
|
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||||
|
margin: 0;
|
||||||
|
padding: 20px;
|
||||||
|
background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
|
||||||
|
min-height: 100vh;
|
||||||
|
}
|
||||||
|
.container {
|
||||||
|
max-width: 1200px;
|
||||||
|
margin: 0 auto;
|
||||||
|
background: white;
|
||||||
|
border-radius: 8px;
|
||||||
|
box-shadow: 0 4px 6px rgba(0,0,0,0.1);
|
||||||
|
padding: 30px;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
color: #2c3e50;
|
||||||
|
text-align: center;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
font-size: 28px;
|
||||||
|
}
|
||||||
|
.subtitle {
|
||||||
|
text-align: center;
|
||||||
|
color: #7f8c8d;
|
||||||
|
margin-bottom: 30px;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
.diagram-wrapper {
|
||||||
|
margin: 30px 0;
|
||||||
|
padding: 28px;
|
||||||
|
background: #f8f9fa;
|
||||||
|
border-radius: 6px;
|
||||||
|
border-left: 4px solid #3498db;
|
||||||
|
overflow-x: auto;
|
||||||
|
}
|
||||||
|
.mermaid {
|
||||||
|
display: flex;
|
||||||
|
justify-content: center;
|
||||||
|
min-width: 760px;
|
||||||
|
}
|
||||||
|
.mermaid svg {
|
||||||
|
width: 100%;
|
||||||
|
max-width: 920px;
|
||||||
|
height: auto;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
.mermaid .nodeLabel,
|
||||||
|
.mermaid .edgeLabel {
|
||||||
|
font-size: 16px !important;
|
||||||
|
}
|
||||||
|
.mermaid .edgeLabel {
|
||||||
|
background: transparent !important;
|
||||||
|
padding: 2px 4px;
|
||||||
|
}
|
||||||
|
.mermaid .edgeLabel:has(span:empty) {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
.legend {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
|
||||||
|
gap: 15px;
|
||||||
|
margin-top: 30px;
|
||||||
|
padding-top: 20px;
|
||||||
|
border-top: 1px solid #ecf0f1;
|
||||||
|
}
|
||||||
|
.legend-item {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
font-size: 13px;
|
||||||
|
}
|
||||||
|
.legend-color {
|
||||||
|
width: 20px;
|
||||||
|
height: 20px;
|
||||||
|
border-radius: 3px;
|
||||||
|
margin-right: 10px;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
.section {
|
||||||
|
margin-top: 40px;
|
||||||
|
padding: 20px;
|
||||||
|
background: #ecf0f1;
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.section h2 {
|
||||||
|
color: #2c3e50;
|
||||||
|
margin-top: 0;
|
||||||
|
font-size: 18px;
|
||||||
|
border-bottom: 2px solid #3498db;
|
||||||
|
padding-bottom: 10px;
|
||||||
|
}
|
||||||
|
.section p {
|
||||||
|
color: #555;
|
||||||
|
line-height: 1.6;
|
||||||
|
margin: 10px 0;
|
||||||
|
}
|
||||||
|
.flow-step {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
margin: 15px 0;
|
||||||
|
padding: 10px;
|
||||||
|
background: white;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
.flow-step span {
|
||||||
|
display: inline-block;
|
||||||
|
width: 110px;
|
||||||
|
min-width: 110px;
|
||||||
|
flex-shrink: 0;
|
||||||
|
font-weight: bold;
|
||||||
|
color: #3498db;
|
||||||
|
}
|
||||||
|
.flow-step p {
|
||||||
|
flex: 1;
|
||||||
|
min-width: 0;
|
||||||
|
}
|
||||||
|
code {
|
||||||
|
background: #f4f4f4;
|
||||||
|
padding: 2px 6px;
|
||||||
|
border-radius: 3px;
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
@keyframes flow {
|
||||||
|
0% { opacity: 0.3; }
|
||||||
|
50% { opacity: 1; }
|
||||||
|
100% { opacity: 0.3; }
|
||||||
|
}
|
||||||
|
.animate-flow {
|
||||||
|
animation: flow 2s ease-in-out infinite;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="container">
|
||||||
|
<h1>Talk-to-Data: Semantic Analytics Architecture</h1>
|
||||||
|
<p class="subtitle">SAP → Snowflake → dbt → Cortex → Chatbot | <1 minute quote answers</p>
|
||||||
|
|
||||||
|
<div class="diagram-wrapper">
|
||||||
|
<div class="mermaid">
|
||||||
|
graph TD
|
||||||
|
SAP["SAP ERP<br/>Quotes and discounts<br/>Daily batch"]
|
||||||
|
RAW["Snowflake raw data"]
|
||||||
|
STG["dbt staging<br/>Clean and standardize"]
|
||||||
|
TRF["dbt transforms<br/>Business rules and flags"]
|
||||||
|
SEMA["Semantic views<br/>Business terms and lineage"]
|
||||||
|
CORTEX["Cortex Analytics<br/>Guardrails and audit"]
|
||||||
|
UI["Snowflake UI<br/>Dashboards"]
|
||||||
|
CHAT["Internal workspace<br/>Chatbot"]
|
||||||
|
USER["Sales and operations<br/>30+ users<br/><1 min estimated answer"]
|
||||||
|
AUDIT["Audit trail<br/>Access and exceptions"]
|
||||||
|
|
||||||
|
SAP --> RAW
|
||||||
|
RAW --> STG
|
||||||
|
STG --> TRF
|
||||||
|
TRF --> SEMA
|
||||||
|
SEMA --> CORTEX
|
||||||
|
CORTEX --> UI
|
||||||
|
CORTEX --> CHAT
|
||||||
|
UI --> USER
|
||||||
|
CHAT --> USER
|
||||||
|
SEMA -.-> AUDIT
|
||||||
|
|
||||||
|
style SAP fill:#d4a574,stroke:#333,stroke-width:2px,color:#fff
|
||||||
|
style CORTEX fill:#ff9999,stroke:#c0392b,stroke-width:2px,color:#fff
|
||||||
|
style SEMA fill:#99ccff,stroke:#2980b9,stroke-width:2px,color:#fff
|
||||||
|
style CHAT fill:#99ff99,stroke:#27ae60,stroke-width:2px,color:#000
|
||||||
|
style USER fill:#ffcc99,stroke:#e67e22,stroke-width:2px,color:#000
|
||||||
|
style AUDIT fill:#cc99ff,stroke:#8e44ad,stroke-width:2px,color:#fff
|
||||||
|
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
|
||||||
|
style STG fill:#d0d7de,stroke:#666,stroke-width:1px
|
||||||
|
style TRF fill:#adb8c1,stroke:#666,stroke-width:1px
|
||||||
|
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="legend">
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #d4a574;"></div>
|
||||||
|
<span><strong>SAP Source:</strong> Daily batch extract</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #e1e4e8;"></div>
|
||||||
|
<span><strong>Raw:</strong> Unmodified data</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #d0d7de;"></div>
|
||||||
|
<span><strong>Staging:</strong> Data preparation</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #adb8c1;"></div>
|
||||||
|
<span><strong>Transform:</strong> Business logic</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #99ccff;"></div>
|
||||||
|
<span><strong>Semantic:</strong> Data contract</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #ff9999;"></div>
|
||||||
|
<span><strong>Cortex:</strong> AI guardrails</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #99ff99;"></div>
|
||||||
|
<span><strong>Chatbot:</strong> User interface</span>
|
||||||
|
</div>
|
||||||
|
<div class="legend-item">
|
||||||
|
<div class="legend-color" style="background: #cc99ff;"></div>
|
||||||
|
<span><strong>Audit:</strong> Compliance trail</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="section">
|
||||||
|
<h2>Data Flow Stages</h2>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>1. Source</span>
|
||||||
|
<p>SAP ERP systems extract quotation, discount, and customer data daily at 05:00 UTC. Data includes quotation identifiers, line items, pricing, and discount conditions.</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>2. Raw</span>
|
||||||
|
<p>Snowflake External Stage ingests CSV/Parquet. Raw schema preserves source structure without modification. Data quality checks validate row counts, null patterns.</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>3. Staging</span>
|
||||||
|
<p><code>stg_*</code> models rename columns, cast data types, and handle nulls. Example: source quotation ID → <code>quote_id</code>, source amount → <code>net_value</code>. No business logic yet.</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>4. Transform</span>
|
||||||
|
<p><code>trf_quotation</code> joins customer policies, product portfolios, and agreement discounts. Calculates discount totals, applies governance flags (e.g., <code>discount_above_ceiling</code>). Two models: <code>trf_quotation</code> (BI) and <code>trf_chatbot_quotation</code> (LLM-optimized).</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>5. Semantic</span>
|
||||||
|
<p>Snowflake DDL semantic views expose curated dimensions and facts. Includes synonyms for natural language (e.g., "desconto cliente" → <code>discount_customer_pct</code>). Single source of truth for BI dashboards and LLM queries.</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>6. Cortex</span>
|
||||||
|
<p><a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst" target="_blank" rel="noopener noreferrer">Cortex Analyst</a> applies guardrails: hides raw discount percentages, protects customer names, checks data freshness, enforces role-based access, and logs all access. LLM queries go through here.</p>
|
||||||
|
</div>
|
||||||
|
<div class="flow-step">
|
||||||
|
<span>7. Consume</span>
|
||||||
|
<p><strong>Snowflake UI:</strong> Dashboards for ops/sales teams (full data visibility). <strong>Internal Workspace:</strong> Chatbot interface (guardrail-filtered data). Both consume the same semantic layer; governance rules ensure consistency.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="section">
|
||||||
|
<h2>Why This Architecture?</h2>
|
||||||
|
<p><strong>Governance-first:</strong> Guardrails (discount policies, protection of personal and customer information, and audit logs) are embedded in the data layer, not bolted on to the LLM prompt. The bot can't bypass policy; it's enforced by Cortex + dbt.</p>
|
||||||
|
<p><strong>Semantic contract:</strong> The semantic layer (sv_quotation, sv_chatbot_quotation) defines the agreement between data engineers, BI teams, and LLM consumers. Column definitions, synonyms, and access rules are version-controlled.</p>
|
||||||
|
<p><strong>Team efficiency:</strong> dbt lineage shows how every dimension flows from SAP → semantic layer. Data engineers and business analysts can trace a discount calculation back to SAP in seconds. Onboarding takes days, not weeks.</p>
|
||||||
|
<p><strong>Prototyping speed:</strong> Snowflake + Cortex means we didn't build custom infrastructure. POC shipped in weeks. Cortex guardrails were ready on day 1; no custom policy engine to build.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="section">
|
||||||
|
<h2>Expected Outcomes</h2>
|
||||||
|
<p><strong>Latency:</strong> Quote research from 5-30 minutes (email + manual lookup) → <1 minute (chatbot query)</p>
|
||||||
|
<p><strong>Scalability:</strong> 30+ salespeople can query the chatbot concurrently. Snowflake handles scale; dbt models are stateless (scale linearly).</p>
|
||||||
|
<p><strong>Governance:</strong> All discount exceptions flagged and audited. Policy violations are visible and traceable.</p>
|
||||||
|
<p><strong>Team capability:</strong> Once-daily batch + semantic layer pattern is reusable. Next use cases (contracts, pricing, serviceability) ship faster by leveraging the same playbook.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="section">
|
||||||
|
<h2>Learn More</h2>
|
||||||
|
<p>
|
||||||
|
See <code>README.md</code> for full narrative, decisions, and lessons learned.
|
||||||
|
Check <code>dbt/</code> for anonymized SQL models.
|
||||||
|
Review <code>cortex/cortex_governance_config.yaml</code> for guardrails and access control.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
14
case_study.css
Normal file
14
case_study.css
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
@page { size: A4; margin: 14mm; }
|
||||||
|
body { font-family: Arial, sans-serif; color: #1f2933; font-size: 10pt; line-height: 1.25; }
|
||||||
|
h1 { color: #0b4f71; font-size: 24pt; margin: 0 0 2pt; }
|
||||||
|
h2 { color: #0b4f71; font-size: 13pt; margin: 10pt 0 4pt; border-bottom: 1px solid #b8c9d3; padding-bottom: 2pt; }
|
||||||
|
h3 { color: #0b4f71; font-size: 11pt; margin: 7pt 0 2pt; }
|
||||||
|
p { margin: 4pt 0; }
|
||||||
|
table { width: 100%; border-collapse: collapse; margin: 4pt 0 7pt; font-size: 9pt; }
|
||||||
|
th { background: #0b4f71; color: white; text-align: left; }
|
||||||
|
th, td { border: 1px solid #cbd5e1; padding: 4pt; vertical-align: top; }
|
||||||
|
code, pre { font-family: monospace; font-size: 8pt; }
|
||||||
|
pre { background: #eef4f7; padding: 6pt; border-left: 3px solid #0b4f71; }
|
||||||
|
ul { margin: 3pt 0 5pt 15pt; padding: 0; }
|
||||||
|
li { margin: 1pt 0; }
|
||||||
|
strong { color: #0b4f71; }
|
||||||
49
case_study_one_page.html
Normal file
49
case_study_one_page.html
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<title>Talk-to-Data Case Study</title>
|
||||||
|
<style>
|
||||||
|
@page{size:A4;margin:14mm}body{font-family:Arial,sans-serif;color:#1f2933;font-size:10pt;line-height:1.25}h1{color:#0b4f71;font-size:24pt;margin:0 0 2pt}h2{color:#0b4f71;font-size:13pt;margin:10pt 0 4pt;border-bottom:1px solid #b8c9d3;padding-bottom:2pt}p{margin:4pt 0}table{width:100%;border-collapse:collapse;margin:4pt 0 7pt;font-size:9pt}th{background:#0b4f71;color:#fff;text-align:left}th,td{border:1px solid #cbd5e1;padding:4pt;vertical-align:top}pre{background:#eef4f7;padding:6pt;border-left:3px solid #0b4f71;font-size:8pt}strong{color:#0b4f71}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>Talk-to-Data</h1>
|
||||||
|
<p><strong>Governance-first semantic analytics for enterprise quotation workflows</strong></p>
|
||||||
|
<h2>The challenge</h2>
|
||||||
|
<p>More than 30 salespeople could depend on internal experts to answer routine quotation questions about products, discounts, and customer agreements. Manual lookup and email or chat coordination introduced an estimated <strong>5–30 minute delay per question</strong>.</p>
|
||||||
|
<h2>The approach</h2>
|
||||||
|
<p>Build a proof of concept around a curated semantic layer instead of exposing raw enterprise tables to an LLM.</p>
|
||||||
|
<pre>SAP ERP
|
||||||
|
│ daily batch
|
||||||
|
▼
|
||||||
|
Snowflake raw → dbt staging → dbt transforms
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Semantic views
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Cortex guardrails & audit
|
||||||
|
┌────────┴────────┐
|
||||||
|
▼ ▼
|
||||||
|
Snowflake UI Internal chatbot</pre>
|
||||||
|
<h2>Architecture decisions</h2>
|
||||||
|
<table><tr><th>Decision</th><th>Why</th></tr>
|
||||||
|
<tr><td>Snowflake + Cortex</td><td>Fast POC path with data, AI, and governance in one platform.</td></tr>
|
||||||
|
<tr><td>dbt layers</td><td>Reproducible SQL, lineage, testing, and versioned business logic.</td></tr>
|
||||||
|
<tr><td>Semantic views</td><td>One business contract for dashboards and natural-language queries.</td></tr>
|
||||||
|
<tr><td>Daily refresh</td><td>Sufficient for quotation research; avoids premature real-time complexity.</td></tr>
|
||||||
|
<tr><td>Guardrails in data layer</td><td>Protection of personal and customer information, access control, freshness checks, and exception escalation are enforceable—not only prompt instructions.</td></tr></table>
|
||||||
|
<h2>Estimated outcome</h2>
|
||||||
|
<table><tr><th>Signal</th><th>Estimate</th></tr>
|
||||||
|
<tr><td>Routine answer latency</td><td><strong>5–30 min → <1 min</strong></td></tr>
|
||||||
|
<tr><td>Sales users in scope</td><td><strong>30+</strong></td></tr>
|
||||||
|
<tr><td>Data freshness</td><td><strong>Daily batch</strong></td></tr>
|
||||||
|
<tr><td>Main benefit</td><td>Less coordination overhead and faster quote responses.</td></tr></table>
|
||||||
|
<p>These are <strong>estimates based on the human workflow</strong>, not measured production KPIs. The next delivery step is a controlled pilot that captures actual latency, adoption, escalation rate, and answer quality.</p>
|
||||||
|
<h2>Delivery lesson</h2>
|
||||||
|
<p>The reusable asset was not just the chatbot. It was the delivery pattern: establish the semantic contract, encode business rules, apply governance, and then expose the smallest useful interface.</p>
|
||||||
|
<p><strong>Stack:</strong> SAP source system · Snowflake · dbt · Snowflake semantic views · Cortex Analytics · internal chatbot workspace</p>
|
||||||
|
<p><em>All examples are anonymized and contain no proprietary source data.</em></p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
55
case_study_one_page.md
Normal file
55
case_study_one_page.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Talk-to-Data
|
||||||
|
## Governance-first semantic analytics for quotation workflows
|
||||||
|
|
||||||
|
### The challenge
|
||||||
|
|
||||||
|
More than 30 salespeople could depend on internal experts to answer routine quotation questions about products, discounts, and customer agreements. Manual lookup and email or chat coordination introduced an estimated **5–30 minute delay per question**.
|
||||||
|
|
||||||
|
### The approach
|
||||||
|
|
||||||
|
Build a proof of concept around a curated semantic layer instead of exposing raw enterprise tables to an LLM.
|
||||||
|
|
||||||
|
```text
|
||||||
|
SAP ERP
|
||||||
|
│ daily batch
|
||||||
|
▼
|
||||||
|
Snowflake raw → dbt staging → dbt transforms
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Semantic views
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Cortex guardrails & audit
|
||||||
|
┌────────┴────────┐
|
||||||
|
▼ ▼
|
||||||
|
Snowflake UI Internal chatbot
|
||||||
|
```
|
||||||
|
|
||||||
|
### Architecture decisions
|
||||||
|
|
||||||
|
| Decision | Why |
|
||||||
|
|---|---|
|
||||||
|
| Snowflake + Cortex | Fast POC path with data, AI, and governance in one platform |
|
||||||
|
| dbt layers | Reproducible SQL, lineage, testing, and versioned business logic |
|
||||||
|
| Semantic views | One business contract for dashboards and natural-language queries |
|
||||||
|
| Daily refresh | Sufficient for the quotation research use case; avoids premature real-time complexity |
|
||||||
|
| Guardrails in the data layer | Protection of personal and customer information, access control, freshness checks, and exception escalation are enforceable—not only prompt instructions |
|
||||||
|
|
||||||
|
### Estimated outcome
|
||||||
|
|
||||||
|
| Signal | Estimate |
|
||||||
|
|---|---:|
|
||||||
|
| Routine answer latency | **5–30 min → <1 min** |
|
||||||
|
| Sales users in scope | **30+** |
|
||||||
|
| Data freshness | **Daily batch** |
|
||||||
|
| Main benefit | Less coordination overhead and faster quote responses |
|
||||||
|
|
||||||
|
These are **estimates based on the human workflow**, not measured production KPIs. The next delivery step is a controlled pilot that captures actual latency, adoption, escalation rate, and answer quality.
|
||||||
|
|
||||||
|
### Delivery lesson
|
||||||
|
|
||||||
|
The reusable asset was not just the chatbot. It was the delivery pattern: establish the semantic contract, encode business rules, apply governance, and then expose the smallest useful interface.
|
||||||
|
|
||||||
|
**Stack:** SAP source system · Snowflake · dbt · Snowflake semantic views · Cortex Analytics · internal chatbot workspace
|
||||||
|
|
||||||
|
*All examples are anonymized and contain no proprietary source data.*
|
||||||
BIN
case_study_one_page.pdf
Normal file
BIN
case_study_one_page.pdf
Normal file
Binary file not shown.
156
cortex_governance_config.yaml
Normal file
156
cortex_governance_config.yaml
Normal file
@@ -0,0 +1,156 @@
|
|||||||
|
# Cortex Analytics Guardrails & Governance Configuration
|
||||||
|
# Semantic Layer: Quotation Analysis
|
||||||
|
# Purpose: Control data access, freshness, and output quality for LLM consumption
|
||||||
|
|
||||||
|
cortex_policies:
|
||||||
|
version: "1.0"
|
||||||
|
semantic_model: "sv_quotation"
|
||||||
|
|
||||||
|
# Data freshness contract: How fresh must data be for different use cases
|
||||||
|
freshness:
|
||||||
|
default_max_age_hours: 24
|
||||||
|
rules:
|
||||||
|
- use_case: "chatbot_quote_inquiry"
|
||||||
|
max_age_hours: 24
|
||||||
|
refresh_trigger: "daily_batch_08_00_utc"
|
||||||
|
description: "Daily batch refresh sufficient for quote research; cutoff 8am UTC (5am EST)"
|
||||||
|
|
||||||
|
- use_case: "real_time_dashboard"
|
||||||
|
max_age_hours: 2
|
||||||
|
refresh_trigger: "hourly"
|
||||||
|
description: "Sales team dashboard requires 2-hour freshness maximum"
|
||||||
|
|
||||||
|
# Access control: Who can query this semantic model
|
||||||
|
access_control:
|
||||||
|
default_role: "[REDACTED: SNOWFLAKE_ROLE]"
|
||||||
|
authorized_teams:
|
||||||
|
- name: "sales_team"
|
||||||
|
snowflake_role: "[REDACTED: SALES_ROLE]"
|
||||||
|
tables: ["sv_quotation", "sv_chatbot_quotation"]
|
||||||
|
max_rows_returned: 100000
|
||||||
|
|
||||||
|
- name: "operations_team"
|
||||||
|
snowflake_role: "[REDACTED: OPS_ROLE]"
|
||||||
|
tables: ["sv_quotation", "sv_chatbot_quotation"]
|
||||||
|
max_rows_returned: 500000
|
||||||
|
|
||||||
|
- name: "cortex_agent"
|
||||||
|
snowflake_role: "[REDACTED: CORTEX_AGENT_ROLE]"
|
||||||
|
tables: ["sv_chatbot_quotation"]
|
||||||
|
max_rows_returned: 1000
|
||||||
|
allowed_functions: ["semantic_search", "similarity_score"]
|
||||||
|
|
||||||
|
# LLM output guardrails: Control what the AI model can do with the data
|
||||||
|
llm_guardrails:
|
||||||
|
|
||||||
|
- rule: "discount_policy_redaction"
|
||||||
|
description: "Hide raw discount values from end users; show only 'within policy' or 'requires approval'"
|
||||||
|
pattern: "discount_.*_pct"
|
||||||
|
action: "redact_numeric_values"
|
||||||
|
replacement_logic: |
|
||||||
|
if discount_total_pct <= discount_ceiling_pct then
|
||||||
|
"Within policy"
|
||||||
|
else
|
||||||
|
"Requires manager approval"
|
||||||
|
end if
|
||||||
|
impact: "User sees governance status, not raw discount rules"
|
||||||
|
|
||||||
|
- rule: "customer_name_masking"
|
||||||
|
description: "Mask actual customer names in chatbot responses"
|
||||||
|
pattern: "customer_name|cliente_nome"
|
||||||
|
action: "mask_value"
|
||||||
|
replacement: "[REDACTED: Customer Information]"
|
||||||
|
exception: "Sales team in authorized_teams can see unmasked values"
|
||||||
|
|
||||||
|
- rule: "pii_scrubbing"
|
||||||
|
description: "Remove personally identifiable information (contact names, emails, phone)"
|
||||||
|
pattern: "sales_person|email|phone|contact_name"
|
||||||
|
action: "redact"
|
||||||
|
exception: "Internal sales team dashboard only"
|
||||||
|
|
||||||
|
- rule: "discount_exception_escalation"
|
||||||
|
description: "Flag when chatbot encounters discount above policy ceiling"
|
||||||
|
pattern: "requires_exception_approval = 1"
|
||||||
|
action: "escalate_to_human"
|
||||||
|
notification: "Send to operations team for manual review"
|
||||||
|
max_escalations_per_hour: 10
|
||||||
|
description: "Prevent bot from auto-approving exceptions"
|
||||||
|
|
||||||
|
# Quality gates: Validation rules before response generation
|
||||||
|
quality_gates:
|
||||||
|
|
||||||
|
- gate: "data_completeness"
|
||||||
|
check: "All required dimensions present (quote_id, material_id, customer_id)"
|
||||||
|
failure_action: "return_error_to_user"
|
||||||
|
error_message: "Quote data incomplete. Please provide quote number."
|
||||||
|
|
||||||
|
- gate: "data_staleness"
|
||||||
|
check: "Data age < freshness.max_age_hours"
|
||||||
|
failure_action: "warn_user"
|
||||||
|
warning_message: "Quote data may be up to 24 hours old."
|
||||||
|
|
||||||
|
- gate: "output_relevance"
|
||||||
|
check: "Cortex confidence score > 0.7 on semantic match"
|
||||||
|
failure_action: "escalate_to_human"
|
||||||
|
threshold_score: 0.7
|
||||||
|
description: "Only respond if model is confident about context"
|
||||||
|
|
||||||
|
# Cost controls: Prevent runaway LLM usage
|
||||||
|
cost_controls:
|
||||||
|
monthly_budget_usd: "[REDACTED: BUDGET]"
|
||||||
|
alert_threshold_pct: 80
|
||||||
|
per_query_max_tokens: 2000
|
||||||
|
max_concurrent_queries: 10
|
||||||
|
|
||||||
|
# Audit & compliance: Track all access and LLM decisions
|
||||||
|
audit:
|
||||||
|
log_level: "full"
|
||||||
|
events_logged:
|
||||||
|
- "user_query"
|
||||||
|
- "data_accessed"
|
||||||
|
- "llm_response_generated"
|
||||||
|
- "redaction_applied"
|
||||||
|
- "exception_escalated"
|
||||||
|
retention_days: 90
|
||||||
|
compliance_flags:
|
||||||
|
- "discount_policy_violation"
|
||||||
|
- "unauthorized_access_attempt"
|
||||||
|
- "data_freshness_breach"
|
||||||
|
|
||||||
|
semantic_model_lineage:
|
||||||
|
description: "How quotation data flows through transformations"
|
||||||
|
stages:
|
||||||
|
1_source:
|
||||||
|
system: "[REDACTED: SAP_SYSTEM]"
|
||||||
|
frequency: "Daily batch 05:00 UTC"
|
||||||
|
tables: ["[REDACTED: SOURCE_QUOTE_TABLE]", "[REDACTED: SOURCE_DISCOUNT_TABLE]"]
|
||||||
|
|
||||||
|
2_staging:
|
||||||
|
schema: "STAGING"
|
||||||
|
models: ["stg_quotes.sql", "stg_discount_conditions.sql"]
|
||||||
|
transformations: "Column renaming, type casting, null handling"
|
||||||
|
|
||||||
|
3_transform:
|
||||||
|
schema: "TRANSFORM"
|
||||||
|
models: ["trf_quotation.sql", "trf_chatbot_quotation.sql"]
|
||||||
|
transformations: "Discount calculation, portfolio mapping, governance flags"
|
||||||
|
|
||||||
|
4_semantic_layer:
|
||||||
|
schema: "DISTRIBUTE_DDL"
|
||||||
|
models: ["sv_quotation", "sv_chatbot_quotation"]
|
||||||
|
purpose: "Semantic views expose business logic, hide implementation details"
|
||||||
|
|
||||||
|
5_cortex_consumption:
|
||||||
|
interface_1: "Snowflake Native App (BI)"
|
||||||
|
interface_2: "[REDACTED: SIEMENS_WORKSPACE]"
|
||||||
|
llm_model: "[REDACTED: LLM_VERSION]"
|
||||||
|
prompt_template: |
|
||||||
|
You are a sales support assistant. Answer quote questions using only
|
||||||
|
the data provided. If discount exceeds policy, flag for human review.
|
||||||
|
Do not disclose raw discount rules.
|
||||||
|
|
||||||
|
notes:
|
||||||
|
- "This configuration enforces 'semantic layer first' — governance lives in data, not prompt engineering."
|
||||||
|
- "Redaction rules + access control + LLM guardrails work together: one fails, user sees safe fallback."
|
||||||
|
- "Cortex policies are version-controlled; audit trail shows all policy changes and who approved them."
|
||||||
|
- "Refresh schedule validates data freshness; chatbot rejects queries on stale data."
|
||||||
43
linkedin_post.md
Normal file
43
linkedin_post.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
# LinkedIn post draft
|
||||||
|
|
||||||
|
Sales teams should not need an email chain to answer a routine quotation question.
|
||||||
|
|
||||||
|
In a proof of concept, I designed a governance-first semantic analytics flow for quotation data:
|
||||||
|
|
||||||
|
**SAP → Snowflake → dbt → Semantic Views → Cortex → Chatbot**
|
||||||
|
|
||||||
|
The business problem was simple: more than 30 salespeople could need help validating a quote, discount, or product classification. A question that looked small could take 5–30 minutes because the answer depended on manual lookup and internal communication.
|
||||||
|
|
||||||
|
The solution was not “put an LLM on top of raw tables.”
|
||||||
|
|
||||||
|
I first created a semantic contract:
|
||||||
|
|
||||||
|
- curated quotation dimensions and facts;
|
||||||
|
- business-friendly synonyms for natural-language questions;
|
||||||
|
- centralized discount calculations;
|
||||||
|
- explicit policy-exception flags;
|
||||||
|
- lineage from source data to the consumer-facing model.
|
||||||
|
|
||||||
|
Then Cortex applied the controls around that contract:
|
||||||
|
|
||||||
|
- data freshness warning for the daily batch;
|
||||||
|
- role-based access;
|
||||||
|
- protection of sensitive information;
|
||||||
|
- escalation when a discount exceeded policy;
|
||||||
|
- audit logging for queries and decisions.
|
||||||
|
|
||||||
|
The semantic model was consumed in two ways:
|
||||||
|
|
||||||
|
1. Snowflake's interface for analytical exploration.
|
||||||
|
2. An internal chatbot workspace for direct questions, with prompt engineering to refine the response experience.
|
||||||
|
|
||||||
|
The estimated user outcome was reducing routine quote research from 5–30 minutes to less than one minute. That is an estimate based on the existing human workflow, not a production benchmark—and that distinction matters.
|
||||||
|
|
||||||
|
The main lesson: **governance belongs in the data and semantic layers, not only in the prompt.**
|
||||||
|
|
||||||
|
The POC also created a reusable delivery pattern for future use cases: define the business contract, expose only the right data, add guardrails, then choose the lightest useful interface.
|
||||||
|
|
||||||
|
I documented the anonymized architecture, decisions, and lessons learned here:
|
||||||
|
[GitHub repository link]
|
||||||
|
|
||||||
|
#DataArchitecture #Snowflake #dbt #DataGovernance #EnterpriseAI #TechnicalDelivery
|
||||||
165
mart_quotes_semantic.yml
Normal file
165
mart_quotes_semantic.yml
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
{{
|
||||||
|
config(
|
||||||
|
materialized='snowflake_semantic_view' )
|
||||||
|
}}
|
||||||
|
name: SV_CUSTOMER_QUOTATION_SEMANTIC
|
||||||
|
description: Semantic view for customer quotation analysis via AI-powered chatbot
|
||||||
|
|
||||||
|
tables:
|
||||||
|
- name: trf_quote_mart
|
||||||
|
description: Customer quotation items with product portfolio classification and sales team metadata. Designed for LLM consumption via chatbot and BI interfaces.
|
||||||
|
base_table: {{ ref('trf_quote_mart') }}
|
||||||
|
|
||||||
|
dimensions:
|
||||||
|
- name: quote_id
|
||||||
|
description: Unique quotation identifier. Links quotation header to line items.
|
||||||
|
expr: quote_id
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- quotation number
|
||||||
|
- quote number
|
||||||
|
- document number
|
||||||
|
|
||||||
|
- name: quote_line_item
|
||||||
|
description: Line item sequence within quotation.
|
||||||
|
expr: quote_line_item
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- item number
|
||||||
|
- line number
|
||||||
|
|
||||||
|
- name: portfolio_code
|
||||||
|
description: Product portfolio classification for grouping related offerings.
|
||||||
|
expr: portfolio_code
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- product category
|
||||||
|
- portfolio element
|
||||||
|
- offering type
|
||||||
|
|
||||||
|
- name: portfolio_name
|
||||||
|
description: Human-readable portfolio name (e.g., Data Integration, Automation, Security).
|
||||||
|
expr: portfolio_name
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- portfolio name
|
||||||
|
- offering name
|
||||||
|
- product line
|
||||||
|
|
||||||
|
- name: material_code
|
||||||
|
description: Product/service identifier. Maps to catalog in source system.
|
||||||
|
expr: material_code
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- product code
|
||||||
|
- material number
|
||||||
|
- SKU
|
||||||
|
|
||||||
|
- name: material_description
|
||||||
|
description: Product/service short name for quotation display.
|
||||||
|
expr: material_description
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- product description
|
||||||
|
- offering description
|
||||||
|
|
||||||
|
- name: customer_id
|
||||||
|
description: Customer account identifier.
|
||||||
|
expr: customer_id
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- customer code
|
||||||
|
- account number
|
||||||
|
|
||||||
|
- name: customer_name
|
||||||
|
description: Customer account name.
|
||||||
|
expr: customer_name
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- account name
|
||||||
|
- customer name
|
||||||
|
|
||||||
|
- name: sales_person
|
||||||
|
description: Sales representative responsible for quotation.
|
||||||
|
expr: sales_person
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- salesperson
|
||||||
|
- account owner
|
||||||
|
- representative
|
||||||
|
|
||||||
|
- name: quote_date
|
||||||
|
description: Quotation creation date.
|
||||||
|
expr: quote_date
|
||||||
|
data_type: TEXT
|
||||||
|
synonyms:
|
||||||
|
- creation date
|
||||||
|
- date
|
||||||
|
- issued date
|
||||||
|
|
||||||
|
facts:
|
||||||
|
- name: quantity
|
||||||
|
description: Quantity of product/service quoted.
|
||||||
|
expr: quantity
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- qty
|
||||||
|
- volume
|
||||||
|
|
||||||
|
- name: net_value
|
||||||
|
description: Net value of quotation line item (after base pricing, before discounts).
|
||||||
|
expr: net_value
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- price
|
||||||
|
- value
|
||||||
|
- amount
|
||||||
|
|
||||||
|
- name: discount_hierarchy_pct
|
||||||
|
description: Hierarchy-based discount percentage (based on customer category).
|
||||||
|
expr: discount_hierarchy_pct
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- hierarchy discount
|
||||||
|
- tier discount
|
||||||
|
|
||||||
|
- name: discount_customer_pct
|
||||||
|
description: Customer-specific negotiated discount percentage.
|
||||||
|
expr: discount_customer_pct
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- customer discount
|
||||||
|
- negotiated discount
|
||||||
|
|
||||||
|
- name: discount_agreement_pct
|
||||||
|
description: Agreement or contract-based discount percentage.
|
||||||
|
expr: discount_agreement_pct
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- agreement discount
|
||||||
|
- contract discount
|
||||||
|
|
||||||
|
- name: discount_total_pct
|
||||||
|
description: Total applied discount across all conditions.
|
||||||
|
expr: discount_total_pct
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- total discount
|
||||||
|
- effective discount
|
||||||
|
|
||||||
|
- name: discount_ceiling_pct
|
||||||
|
description: Maximum allowed discount policy ceiling.
|
||||||
|
expr: discount_ceiling_pct
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- discount limit
|
||||||
|
- max discount
|
||||||
|
- LOA ceiling
|
||||||
|
|
||||||
|
- name: discount_above_ceiling
|
||||||
|
description: Percentage points of discount exceeding policy ceiling (governance flag).
|
||||||
|
expr: discount_above_ceiling
|
||||||
|
data_type: NUMBER
|
||||||
|
synonyms:
|
||||||
|
- above limit
|
||||||
|
- exception discount
|
||||||
113
marts_quotes.sql
Normal file
113
marts_quotes.sql
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
{{
|
||||||
|
config(
|
||||||
|
materialized='table',
|
||||||
|
database=target.database,
|
||||||
|
schema='transform',
|
||||||
|
tags=['semantic_layer', 'chatbot']
|
||||||
|
)
|
||||||
|
}}
|
||||||
|
|
||||||
|
with source_quotes as (
|
||||||
|
select
|
||||||
|
[REDACTED: quote_id column],
|
||||||
|
[REDACTED: line_item column],
|
||||||
|
[REDACTED: customer_id column],
|
||||||
|
[REDACTED: material_id column],
|
||||||
|
[REDACTED: quantity column],
|
||||||
|
[REDACTED: net_value column],
|
||||||
|
[REDACTED: quote_date column],
|
||||||
|
[REDACTED: sales_person_id column],
|
||||||
|
[REDACTED: source_timestamp column]
|
||||||
|
from [REDACTED: SOURCE_SCHEMA].[REDACTED: QUOTE_TABLE]
|
||||||
|
where [REDACTED: date_partition] >= dateadd(day, -1, current_date())
|
||||||
|
),
|
||||||
|
|
||||||
|
enrich_product_portfolio as (
|
||||||
|
select
|
||||||
|
sq.quote_id,
|
||||||
|
sq.line_item,
|
||||||
|
sq.customer_id,
|
||||||
|
sq.material_id,
|
||||||
|
sq.quantity,
|
||||||
|
sq.net_value,
|
||||||
|
sq.quote_date,
|
||||||
|
sq.sales_person_id,
|
||||||
|
/* Product portfolio classification */
|
||||||
|
case
|
||||||
|
when sq.material_id like '[REDACTED: pattern 1]%' then 'data-integration'
|
||||||
|
when sq.material_id like '[REDACTED: pattern 2]%' then 'automation'
|
||||||
|
when sq.material_id like '[REDACTED: pattern 3]%' then 'connectivity'
|
||||||
|
else 'other'
|
||||||
|
end as portfolio_code,
|
||||||
|
pm.portfolio_name,
|
||||||
|
pm.portfolio_category,
|
||||||
|
pm.material_description
|
||||||
|
from source_quotes sq
|
||||||
|
left join {{ ref('dim_product_master') }} pm
|
||||||
|
on sq.material_id = pm.material_id
|
||||||
|
),
|
||||||
|
|
||||||
|
calculate_discounts as (
|
||||||
|
select
|
||||||
|
epf.*,
|
||||||
|
/* Discount hierarchy: customer segment -> discount tier */
|
||||||
|
coalesce(cd.discount_tier_pct, 0) as discount_hierarchy_pct,
|
||||||
|
/* Customer-specific negotiated discount */
|
||||||
|
coalesce(cd.customer_discount_pct, 0) as discount_customer_pct,
|
||||||
|
/* Agreement-based discounts (volume, multi-year, etc.) */
|
||||||
|
coalesce(ca.agreement_discount_pct, 0) as discount_agreement_pct,
|
||||||
|
/* Maximum allowed discount (governance constraint) */
|
||||||
|
cd.discount_ceiling_pct,
|
||||||
|
/* Total discount calculation */
|
||||||
|
coalesce(cd.discount_tier_pct, 0) +
|
||||||
|
coalesce(cd.customer_discount_pct, 0) +
|
||||||
|
coalesce(ca.agreement_discount_pct, 0) as discount_total_pct
|
||||||
|
from enrich_product_portfolio epf
|
||||||
|
left join {{ ref('dim_customer_discount_policy') }} cd
|
||||||
|
on epf.customer_id = cd.customer_id
|
||||||
|
and epf.portfolio_code = cd.portfolio_code
|
||||||
|
left join {{ ref('fct_agreement_discounts') }} ca
|
||||||
|
on epf.customer_id = ca.customer_id
|
||||||
|
and epf.material_id = ca.material_id
|
||||||
|
),
|
||||||
|
|
||||||
|
governance_flags as (
|
||||||
|
select
|
||||||
|
cd.*,
|
||||||
|
/* Flag: discount exceeds policy maximum */
|
||||||
|
case
|
||||||
|
when cd.discount_total_pct > cd.discount_ceiling_pct
|
||||||
|
then cd.discount_total_pct - cd.discount_ceiling_pct
|
||||||
|
else 0
|
||||||
|
end as discount_above_ceiling,
|
||||||
|
/* Governance indicator: quote ready for human review */
|
||||||
|
case
|
||||||
|
when cd.discount_total_pct > cd.discount_ceiling_pct then 1
|
||||||
|
else 0
|
||||||
|
end as requires_exception_approval
|
||||||
|
from calculate_discounts cd
|
||||||
|
)
|
||||||
|
|
||||||
|
select
|
||||||
|
quote_id,
|
||||||
|
line_item,
|
||||||
|
customer_id,
|
||||||
|
material_id,
|
||||||
|
portfolio_code,
|
||||||
|
portfolio_name,
|
||||||
|
material_description,
|
||||||
|
quantity,
|
||||||
|
net_value,
|
||||||
|
discount_hierarchy_pct,
|
||||||
|
discount_customer_pct,
|
||||||
|
discount_agreement_pct,
|
||||||
|
discount_total_pct,
|
||||||
|
discount_ceiling_pct,
|
||||||
|
discount_above_ceiling,
|
||||||
|
requires_exception_approval,
|
||||||
|
quote_date,
|
||||||
|
sales_person_id,
|
||||||
|
current_timestamp() as dbt_loaded_at
|
||||||
|
from governance_flags
|
||||||
|
|
||||||
|
-- ponytail: materialized as table for chatbot + BI queries (not incremental yet; append-only refresh if needed later)
|
||||||
Reference in New Issue
Block a user