269 lines
17 KiB
Markdown
269 lines
17 KiB
Markdown
# Talk-to-Data: Semantic Analytics for Enterprise Quotations
|
||
|
||
**A PoC case study in governance-first semantic layer design: SAP → Snowflake → Cortex → Chatbot.**
|
||
|
||
---
|
||
|
||
## Problem Statement
|
||
|
||
Sales and operations teams across multiple regions handle thousands of quote requests monthly. Each inquiry requires:
|
||
- **Manual research**: Discount policy lookups, product classification, customer contract data
|
||
- **Cross-system queries**: SAP for pricing, internal databases for agreements, email chains with ops
|
||
- **Coordination overhead**: Sales emails ops, ops researches and replies (5-30 minutes per quote)
|
||
- **Scale challenge**: 30+ salespersons generating inquiries daily across the sales organization
|
||
|
||
**Result**: Quote turnaround times of 5-30 minutes, even for routine requests. Ops team context-switches between email, SAP, and spreadsheets. Sales can't get answers fast enough to close deals.
|
||
|
||
---
|
||
|
||
## Solution: Semantic Analytics Layer + Cortex Guardrails
|
||
|
||
**Core insight**: Build a semantic layer that serves as the single source of truth for quote logic, then consume it via two interfaces: BI dashboards and an LLM-powered chatbot. Governance lives in the data layer, not in prompts.
|
||
|
||
### Architecture Overview
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ SAP ERP (Sales, Pricing, Discounts) │
|
||
│ Updated daily via batch extract (05:00 UTC) │
|
||
└────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
Daily ELT Job
|
||
│
|
||
┌────────────────────────▼────────────────────────────────────┐
|
||
│ Snowflake Raw Layer │
|
||
│ - raw.quotations (document, line item, amount, ...) │
|
||
│ - raw.discount_conditions (discount tiers, policies) │
|
||
│ - raw.customer_master (customer codes, agreements) │
|
||
└────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
dbt Staging
|
||
│
|
||
┌────────────────────────▼────────────────────────────────────┐
|
||
│ Staging Layer (STAGING schema) │
|
||
│ - Column mapping, type casting, null handling │
|
||
│ - No business logic, just data preparation │
|
||
└────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
dbt Transforms
|
||
│
|
||
┌────────────────────────▼────────────────────────────────────┐
|
||
│ Transform Layer (TRANSFORM schema) │
|
||
│ - Join discount conditions, policies, agreements │
|
||
│ - Calculate: discount tiers, totals, exceptions │
|
||
│ - Apply: governance flags, audit columns │
|
||
│ - Models: trf_quotation, trf_chatbot_quotation │
|
||
└────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
Semantic Views (DDL)
|
||
│
|
||
┌────────────────────────▼────────────────────────────────────┐
|
||
│ Semantic Layer (DISTRIBUTE_DDL schema) │
|
||
│ - sv_quotation: Full BI consumption │
|
||
│ - sv_chatbot_quotation: LLM-optimized dimensions │
|
||
│ - Exposes: business logic (discounts, policies) │
|
||
│ - Hides: implementation details (temp tables, keys) │
|
||
└────────────────────────┬────────────────────────────────────┘
|
||
│
|
||
Cortex Analytics Guardrails
|
||
│
|
||
┌───────────────┴───────────────┐
|
||
│ │
|
||
┌────────▼──────────────┐ ┌─────────▼─────────────┐
|
||
│ Snowflake UI │ │ Internal Workspace │
|
||
│ (BI Dashboard) │ │ (Chatbot + LLM) │
|
||
│ │ │ │
|
||
│ - Visual analysis │ │ - Quote Q&A │
|
||
│ - Sales reporting │ │ - Prompt engineering │
|
||
│ - Ops monitoring │ │ - Real-time answers │
|
||
└───────────┬───── ─────┘ └──────────┬────────────┘
|
||
│ │
|
||
│ <1 minute turnaround
|
||
│
|
||
End Users (Sales, Ops)
|
||
```
|
||
|
||
---
|
||
|
||
## Stack: Why Each Layer?
|
||
|
||
### SAP → Snowflake ELT (Daily Batch)
|
||
- **Why**: Quotation data lives in SAP; Snowflake is the reporting warehouse
|
||
- **How**: Daily batch extract (05:00 UTC), once-per-day refresh sufficient for quote workflow (real-time inventory not required)
|
||
- **Trade-off**: Once-daily freshness OK for most quote requests; could escalate to hourly for real-time dashboard if needed
|
||
|
||
### Staging → Transform → Semantic Layer (dbt)
|
||
- **Why dbt?**
|
||
- **Lineage**: Column-level traceability (SAP field → transform → semantic dimension)
|
||
- **Testability**: Unit tests on discount calculations, data quality checks
|
||
- **Team reuse**: SQL-first approach, skill portability across data team
|
||
- **Governance**: Schema.yml documents business logic and access rules
|
||
- **Reproducibility**: Version-controlled transformations, audit trail of changes
|
||
|
||
- **Why separate layers?**
|
||
- **Staging**: Decouples source-system brittleness from business logic
|
||
- **Transform**: Concentrates business rules (discount math, policy flags) in one place
|
||
- **Semantic**: Exposes curated dimensions to downstream consumers; hides technical debt
|
||
|
||
### Snowflake Semantic Views
|
||
- **What**: Semantic views (DDL) wrap transform models; define dimensions, facts, and synonyms
|
||
- **Why**:
|
||
- Define the contract between data and consumers (Sales, Cortex)
|
||
- Synonyms enable natural-language queries ("desconto cliente" → `discount_customer_pct`)
|
||
- Consistent schema across BI dashboards and LLM consumption
|
||
- **Example**: A chatbot user asks "What's the customer discount on this quote?" The semantic layer maps "customer discount" (natural language) to `discount_customer_pct` (column), and Cortex applies guardrails (hides the raw percentage and shows only "within policy" or "requires approval")
|
||
|
||
### Cortex Analytics + Guardrails
|
||
- **What**: Snowflake's native LLM platform + custom governance policies
|
||
- **Why Cortex, not custom LLM?**
|
||
- Built-in guardrails: protection of sensitive information, access control, and cost controls
|
||
- Fast prototyping: Cortex agents consume Snowflake semantic views directly (no ETL to another system)
|
||
- Governance-as-code: Policies live alongside data, versioned, audited
|
||
- Security: Data stays in Snowflake; LLM calls don't leak sensitive data
|
||
- **Trade-off**: Cortex is Snowflake-specific (vs. generic LLM + Anthropic); acceptable for POC, revisit if multi-cloud needed later
|
||
|
||
### Dual Consumption (Snowflake UI + Internal Workspace)
|
||
- **Snowflake UI**: Native BI dashboards for sales/ops teams. Semantic model feeds Power BI or Tableau directly.
|
||
- **Internal Workspace**: Existing chatbot environment. Cortex queries the semantic layer + prompt engineering layer refines responses.
|
||
- **Why two interfaces?** Users consume data differently — dashboards for exploration, chatbot for direct answers. Semantic layer unified the source, reducing maintenance.
|
||
|
||
---
|
||
|
||
## Key Decisions & Trade-offs
|
||
|
||
| Decision | Rationale | Trade-off |
|
||
|----------|-----------|----------|
|
||
| **Governance-first** | Discount policies, protection of personal and customer information, and exception escalation built into the data layer | Requires upfront effort to define policies; can't skip it for "speed" |
|
||
| **Semantic views (DDL)** | Single contract between data → BI/LLM | Requires mapping business terms to columns; initial work to get synonyms right |
|
||
| **Once-daily batch** | Sufficient for quote workflow; reduces ELT complexity | Not real-time; acceptable for POC; upgrade to hourly if pilot shows need |
|
||
| **Cortex (vs. custom LLM)** | Fast prototyping, built-in guardrails, stays in Snowflake ecosystem | Cortex-only (could integrate with Anthropic later) |
|
||
| **Separate transform model per interface** | `trf_quotation` for BI, `trf_chatbot_quotation` for LLM — optimize each | Slight duplication; worth it if interfaces have different freshness/cardinality needs |
|
||
|
||
---
|
||
|
||
## Governance: How We Built Safety In
|
||
|
||
Governance is not a checklist; it's embedded in the data layer.
|
||
|
||
### 1. **Discount Policy Guardrails**
|
||
- **Rule**: No discount exceeds the configured policy ceiling
|
||
- **Implementation**:
|
||
- Transform model calculates `discount_above_ceiling` flag
|
||
- Cortex policy hides raw discount percentages; shows only "Within Policy" or "Requires Approval"
|
||
- Chatbot escalates flagged quotes to ops team (max 10 escalations/hour to prevent bot spam)
|
||
- **Effect**: Sales can't accidentally propose an unapproved discount; exceptions are visible and auditable
|
||
|
||
### 2. **Protection of Personal and Customer Information**
|
||
- **Rule**: The chatbot must not expose customer names, contact details, or negotiated terms unless the user is authorized
|
||
- **Implementation**:
|
||
- `sv_chatbot_quotation` includes only aggregated customer data (customer_id, portfolio_code)
|
||
- Cortex policy replaces `customer_name` with "[Customer information protected]" in chatbot responses
|
||
- Exception: Sales team in authorized role can see unmasked values via dashboard
|
||
- **Effect**: Chatbot can answer quote questions without leaking customer contracts
|
||
|
||
### 3. **Access Control & Audit Trail**
|
||
- **Rule**: Different roles see different data
|
||
- Sales: Can query `sv_chatbot_quotation` (limited fields, no raw discounts)
|
||
- Ops: Can query `sv_quotation` (full discount details, for policy review)
|
||
- Cortex Agent: Can query `sv_chatbot_quotation` only (locked down for LLM)
|
||
- **Implementation**: Snowflake role-based access control (RBAC) + Cortex `authorized_teams` config
|
||
- **Audit**: All queries logged; policy violations trigger alerts
|
||
|
||
### 4. **Data Freshness Contract**
|
||
- **Rule**: Chatbot rejects queries on stale data; warns user if data is > 24 hours old
|
||
- **Implementation**:
|
||
- dbt model adds `load_timestamp` column
|
||
- Cortex quality gate checks: `current_timestamp() - load_timestamp < 24 hours`
|
||
- Failure: Returns "Data unavailable; please retry in 1 hour" (prevents stale-data surprises)
|
||
|
||
---
|
||
|
||
## Results: From PoC to Impact
|
||
|
||
### Estimated Impact (Based on User Interviews)
|
||
- **Quote turnaround**: 5-30 min (email + manual lookup) → <1 min (chatbot query)
|
||
- Savings per quote: 5-30 minutes
|
||
- Pilots with 5 sales reps: ~50+ hours/week freed (estimated)
|
||
- Extrapolation: 30+ sales team × 5 hours/week = 150+ hours recovered org-wide
|
||
- **Error reduction**: Discount exceptions caught by policy guardrails (vs. manually reviewed)
|
||
- **Team velocity**: PoC shipped in weeks (not months), governance patterns established for scale
|
||
|
||
### Why This Matters for Delivery Managers
|
||
1. **Governance-first approach**: We didn't build a prototype and hope for policy later. Guardrails were baked in from day one.
|
||
2. **Reusable patterns**: The semantic layer + Cortex + dbt playbook works for any quote/contract/discount domain. Next use cases ship faster.
|
||
3. **Measurable before scale**: We validated impact with a small pilot; ready to scale to 30+ users with confidence.
|
||
4. **Ops visibility**: Sales/ops can see what the bot is doing (audit logs, escalation flags); not a black box.
|
||
|
||
---
|
||
|
||
## How to Use This Repo
|
||
|
||
### Deliverables
|
||
- **`dbt/`**: Anonymized dbt models (staging, transforms, semantic views)
|
||
- `models/staging/stg_*.sql` — Raw layer transformations
|
||
- `models/transform/trf_*.sql` — Business logic
|
||
- `models/distribute_ddl/sv_*.yml` — Semantic views with synonyms
|
||
- **`cortex/`**: Governance config (information protection, access control, audit rules)
|
||
- `cortex_governance_config.yaml` — Guardrails, freshness contracts, LLM policies
|
||
- **`docs/`**: Architecture diagrams, decision logs
|
||
- `architecture_dag.md` — Data flow Mermaid diagram
|
||
- `decisions.md` — Architecture decision records (ADRs)
|
||
- **`README.md`**: This file (the narrative for interviewers/stakeholders)
|
||
|
||
### How to Adapt This to Your Domain
|
||
1. **Replace discount logic** with your domain (e.g., pricing tiers, contract terms, approval workflows)
|
||
2. **Adjust freshness** (once-daily to hourly/real-time) based on your use case
|
||
3. **Modify semantic dimensions** to match your business terminology
|
||
4. **Update Cortex policies** for your personal-information and governance rules
|
||
5. **Test with a small pilot** (5-10 users) before org-wide rollout
|
||
|
||
---
|
||
|
||
## Lessons Learned
|
||
|
||
### What Worked
|
||
✅ **Semantic views as the contract**: Defining dimensions + synonyms upfront saved rework later. Sales/ops understood the model; Cortex had clear field mappings.
|
||
|
||
✅ **Governance in the data layer**: Embedding discount flags and information-protection rules in dbt/Cortex meant the bot couldn't bypass them. No workaround hacks possible.
|
||
|
||
✅ **Once-daily batch sufficient**: For this workflow, 24-hour freshness was OK. We didn't need real-time complexity (yet).
|
||
|
||
✅ **dbt for team alignment**: SQL-first + schema docs meant the data team and business stakeholders spoke the same language. Onboarding new people was fast.
|
||
|
||
### What We'd Change at Scale
|
||
⚠️ **Cortex cost**: Monitor Cortex token usage as chatbot volume scales. Current config has cost controls (monthly budget, per-query max tokens); may need aggressive pruning at scale.
|
||
|
||
⚠️ **Semantic view complexity**: If adding more dimensions (15→50), consider splitting into focused semantic views (e.g., `sv_quote_discounts`, `sv_quote_compliance`) to keep queries fast.
|
||
|
||
---
|
||
|
||
## Tech Stack Summary
|
||
|
||
| Component | Tool | Why |
|
||
|-----------|------|-----|
|
||
| **Data Warehouse** | Snowflake | Scalable, built-in Cortex, governance features |
|
||
| **Transformations** | dbt + Jinja | SQL lineage, testability, team skill reuse |
|
||
| **Semantic Layer** | Snowflake DDL | Data contract, synonyms for NLP, single source of truth |
|
||
| **AI/Guardrails** | Cortex Analytics | Native LLM, information protection, access control, cost controls |
|
||
| **Consumption 1** | Snowflake Native App / BI Tool | Dashboard for ops/sales exploration |
|
||
| **Consumption 2** | Internal Workspace + Prompt Engineering | Chatbot for direct question-answering |
|
||
| **Infrastructure** | Cloud-native Snowflake | No ops overhead, pay-per-query |
|
||
|
||
---
|
||
|
||
## FAQ
|
||
|
||
**Q: Can I use this with a different warehouse (e.g., BigQuery, Redshift)?**
|
||
A: Yes. Adapt the dbt dialect (BigQuery: `jinja-sql`, Redshift: `redshift` profile). Cortex is Snowflake-native; substitute with your own LLM guardrails (policy enforcement layer).
|
||
|
||
**Q: Is once-daily refresh really enough?**
|
||
A: For quote research, yes. If you need real-time pricing updates, escalate to hourly ELT. The architecture supports it; just change the schedule in your orchestrator.
|
||
|
||
**Q: What if I need to scale to 500+ users?**
|
||
A: Semantic layer architecture scales horizontally. Snowflake handles concurrency. Monitor Cortex token usage and partition semantic views if queries slow down. dbt stays the same. The real issue comes when you need RLS requirements, this will enforce you to manager with additional tools or methods beyond Snowflake RBAC (Role Based Access Control).
|
||
|
||
**Q: Can I export this as a Snowflake Native App for partners?**
|
||
A: Yes. Package the semantic views + Cortex policies as an app; partners can install it and use the chatbot without seeing raw data. Cortex Analyst can be seamlessly integrated into any application using REST API.
|
||
|
||
---
|