chore: ignore interactive diagram sources
Keep diagram source files local while publishing the static PDF.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@gabriel.pereira
1
.gitignore
vendored
1
.gitignore
vendored
@@ -3,3 +3,4 @@ case_study.css
|
||||
case_study_one_page.html
|
||||
case_study_one_page.md
|
||||
architecture_dag.md
|
||||
architecture_diagram.html
|
||||
|
||||
@@ -1,59 +0,0 @@
|
||||
# Quotation Semantic Layer - Data Flow DAG
|
||||
# SAP Source -> Snowflake -> dbt Transforms -> Semantic Views -> Cortex -> Chatbot Output
|
||||
|
||||
graph LR
|
||||
SAP["🗄️ SAP ERP System<br/>(Sales & Discount Data)"]
|
||||
|
||||
EXT["📥 Snowflake<br/>External Stage"]
|
||||
|
||||
RAW["raw.quotations<br/>raw.discount_conditions"]
|
||||
|
||||
STG["STAGING Layer<br/>stg_quotes<br/>stg_discount_conditions<br/>stg_product_master"]
|
||||
|
||||
TRF["TRANSFORM Layer<br/>trf_quotation<br/>trf_chatbot_quotation"]
|
||||
|
||||
DIM["Reference Dimensions<br/>dim_customer_discount_policy<br/>dim_product_portfolio<br/>fct_agreement_discounts"]
|
||||
|
||||
SEMA["SEMANTIC LAYER<br/>sv_quotation<br/>sv_chatbot_quotation"]
|
||||
|
||||
CORTEX["Cortex Analytics<br/>Guardrails & Information Protection<br/>Access Control<br/>Freshness Contract"]
|
||||
|
||||
UI["Snowflake UI<br/>(BI Dashboard)"]
|
||||
CHAT["🤖 Internal Workspace<br/>(Chatbot)"]
|
||||
|
||||
SAP -->|Daily Batch 05:00 UTC| EXT
|
||||
EXT -->|Load| RAW
|
||||
|
||||
RAW -->|Column mapping<br/>Type casting| STG
|
||||
|
||||
STG -->|Join dimensions<br/>Calculate discounts<br/>Apply governance flags| TRF
|
||||
DIM -->|Enrich with policies| TRF
|
||||
|
||||
TRF -->|Expose semantic<br/>business logic<br/>Hide implementation| SEMA
|
||||
|
||||
SEMA -->|Apply guardrails<br/>Protect sensitive information<br/>Enforce freshness| CORTEX
|
||||
|
||||
CORTEX -->|Query & Visualize| UI
|
||||
CORTEX -->|LLM consumption<br/>Prompt engineering layer| CHAT
|
||||
|
||||
CHAT -->|<1 min<br/>Quote Answers| USER["👤 Sales & Ops<br/>Team"]
|
||||
UI -->|Dashboard<br/>Reporting| OPS["📊 Operations<br/>Team"]
|
||||
|
||||
SEMA -.->|Data Contract<br/>Lineage| AUDIT["🔒 Audit & Compliance<br/>- Access logs<br/>- Policy changes<br/>- Exception escalations"]
|
||||
|
||||
style SAP fill:#d4a574
|
||||
style CORTEX fill:#ff9999
|
||||
style SEMA fill:#99ccff
|
||||
style CHAT fill:#99ff99
|
||||
style USER fill:#ffcc99
|
||||
style AUDIT fill:#cc99ff
|
||||
|
||||
classDef layer_raw fill:#e1e4e8
|
||||
classDef layer_stg fill:#d0d7de
|
||||
classDef layer_trf fill:#adb8c1
|
||||
classDef layer_sema fill:#6e7681
|
||||
|
||||
class RAW layer_raw
|
||||
class STG layer_stg
|
||||
class TRF layer_trf
|
||||
class SEMA layer_sema
|
||||
@@ -1,331 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>Talk-to-Data: Architecture Animated Diagram</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
|
||||
<style>
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
margin: 0;
|
||||
padding: 20px;
|
||||
background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);
|
||||
min-height: 100vh;
|
||||
}
|
||||
.container {
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
border-radius: 8px;
|
||||
box-shadow: 0 4px 6px rgba(0,0,0,0.1);
|
||||
padding: 30px;
|
||||
}
|
||||
h1 {
|
||||
color: #2c3e50;
|
||||
text-align: center;
|
||||
margin-bottom: 10px;
|
||||
font-size: 28px;
|
||||
}
|
||||
.subtitle {
|
||||
text-align: center;
|
||||
color: #7f8c8d;
|
||||
margin-bottom: 30px;
|
||||
font-size: 14px;
|
||||
}
|
||||
.diagram-wrapper {
|
||||
margin: 30px 0;
|
||||
padding: 28px;
|
||||
background: #f8f9fa;
|
||||
border-radius: 6px;
|
||||
border-left: 4px solid #3498db;
|
||||
overflow-x: auto;
|
||||
}
|
||||
.mermaid {
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
min-width: 760px;
|
||||
}
|
||||
.mermaid svg {
|
||||
width: 100%;
|
||||
max-width: 920px;
|
||||
height: auto;
|
||||
overflow: hidden;
|
||||
}
|
||||
.mermaid .nodeLabel,
|
||||
.mermaid .edgeLabel {
|
||||
font-size: 16px !important;
|
||||
}
|
||||
.mermaid .edgeLabel {
|
||||
background: transparent !important;
|
||||
padding: 2px 4px;
|
||||
}
|
||||
.mermaid .edgeLabel:has(span:empty) {
|
||||
display: none;
|
||||
}
|
||||
.legend {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
|
||||
gap: 15px;
|
||||
margin-top: 30px;
|
||||
padding-top: 20px;
|
||||
border-top: 1px solid #ecf0f1;
|
||||
}
|
||||
.legend-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
font-size: 13px;
|
||||
}
|
||||
.legend-color {
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
border-radius: 3px;
|
||||
margin-right: 10px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.section {
|
||||
margin-top: 40px;
|
||||
padding: 20px;
|
||||
background: #ecf0f1;
|
||||
border-radius: 6px;
|
||||
}
|
||||
.section h2 {
|
||||
color: #2c3e50;
|
||||
margin-top: 0;
|
||||
font-size: 18px;
|
||||
border-bottom: 2px solid #3498db;
|
||||
padding-bottom: 10px;
|
||||
}
|
||||
.section p {
|
||||
color: #555;
|
||||
line-height: 1.6;
|
||||
margin: 10px 0;
|
||||
}
|
||||
.flow-step {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
margin: 15px 0;
|
||||
padding: 10px;
|
||||
background: white;
|
||||
border-radius: 4px;
|
||||
}
|
||||
.flow-step span {
|
||||
display: inline-block;
|
||||
width: 110px;
|
||||
min-width: 110px;
|
||||
flex-shrink: 0;
|
||||
font-weight: bold;
|
||||
color: #3498db;
|
||||
}
|
||||
.flow-step p {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
code {
|
||||
background: #f4f4f4;
|
||||
padding: 2px 6px;
|
||||
border-radius: 3px;
|
||||
font-size: 12px;
|
||||
}
|
||||
@keyframes flow {
|
||||
0% { opacity: 0.3; }
|
||||
50% { opacity: 1; }
|
||||
100% { opacity: 0.3; }
|
||||
}
|
||||
.animate-flow {
|
||||
animation: flow 2s ease-in-out infinite;
|
||||
}
|
||||
@media print {
|
||||
@page {
|
||||
size: A4 portrait;
|
||||
margin: 12mm;
|
||||
}
|
||||
body {
|
||||
padding: 0;
|
||||
background: white;
|
||||
}
|
||||
.container {
|
||||
max-width: none;
|
||||
padding: 0;
|
||||
box-shadow: none;
|
||||
}
|
||||
h1 {
|
||||
font-size: 22px;
|
||||
margin: 0 0 4px;
|
||||
}
|
||||
.subtitle {
|
||||
margin: 0 0 12px;
|
||||
font-size: 11px;
|
||||
}
|
||||
.diagram-wrapper {
|
||||
margin: 0;
|
||||
padding: 8px;
|
||||
border-left-width: 3px;
|
||||
break-inside: avoid;
|
||||
}
|
||||
.mermaid {
|
||||
min-width: 0;
|
||||
}
|
||||
.mermaid svg {
|
||||
width: auto;
|
||||
max-width: 100%;
|
||||
max-height: 185mm;
|
||||
}
|
||||
.legend {
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
gap: 6px;
|
||||
margin-top: 10px;
|
||||
padding-top: 8px;
|
||||
font-size: 9px;
|
||||
break-inside: avoid;
|
||||
}
|
||||
.legend-item {
|
||||
font-size: 9px;
|
||||
}
|
||||
.legend-color {
|
||||
width: 12px;
|
||||
height: 12px;
|
||||
margin-right: 5px;
|
||||
}
|
||||
.section {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>Talk-to-Data: Semantic Analytics Architecture</h1>
|
||||
<p class="subtitle">SAP → Snowflake → dbt → Cortex → Chatbot | <1 minute quote answers</p>
|
||||
|
||||
<div class="diagram-wrapper">
|
||||
<div class="mermaid">
|
||||
graph TD
|
||||
SAP["SAP ERP<br/>Quotes and discounts<br/>Daily batch"]
|
||||
RAW["Snowflake raw data"]
|
||||
STG["dbt staging<br/>Clean and standardize"]
|
||||
TRF["dbt transforms<br/>Business rules and flags"]
|
||||
SEMA["Semantic views<br/>Business terms and lineage"]
|
||||
CORTEX["Cortex Analytics<br/>Guardrails and audit"]
|
||||
UI["Snowflake UI<br/>Dashboards"]
|
||||
CHAT["Internal workspace<br/>Chatbot"]
|
||||
USER["Sales and operations<br/>30+ users<br/><1 min estimated answer"]
|
||||
AUDIT["Audit trail<br/>Access and exceptions"]
|
||||
|
||||
SAP --> RAW
|
||||
RAW --> STG
|
||||
STG --> TRF
|
||||
TRF --> SEMA
|
||||
SEMA --> CORTEX
|
||||
CORTEX --> UI
|
||||
CORTEX --> CHAT
|
||||
UI --> USER
|
||||
CHAT --> USER
|
||||
SEMA -.-> AUDIT
|
||||
|
||||
style SAP fill:#d4a574,stroke:#333,stroke-width:2px,color:#fff
|
||||
style CORTEX fill:#ff9999,stroke:#c0392b,stroke-width:2px,color:#fff
|
||||
style SEMA fill:#99ccff,stroke:#2980b9,stroke-width:2px,color:#fff
|
||||
style CHAT fill:#99ff99,stroke:#27ae60,stroke-width:2px,color:#000
|
||||
style USER fill:#ffcc99,stroke:#e67e22,stroke-width:2px,color:#000
|
||||
style AUDIT fill:#cc99ff,stroke:#8e44ad,stroke-width:2px,color:#fff
|
||||
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
|
||||
style STG fill:#d0d7de,stroke:#666,stroke-width:1px
|
||||
style TRF fill:#adb8c1,stroke:#666,stroke-width:1px
|
||||
style RAW fill:#e1e4e8,stroke:#666,stroke-width:1px
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="legend">
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #d4a574;"></div>
|
||||
<span><strong>SAP Source:</strong> Daily batch extract</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #e1e4e8;"></div>
|
||||
<span><strong>Raw:</strong> Unmodified data</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #d0d7de;"></div>
|
||||
<span><strong>Staging:</strong> Data preparation</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #adb8c1;"></div>
|
||||
<span><strong>Transform:</strong> Business logic</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #99ccff;"></div>
|
||||
<span><strong>Semantic:</strong> Data contract</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #ff9999;"></div>
|
||||
<span><strong>Cortex:</strong> AI guardrails</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #99ff99;"></div>
|
||||
<span><strong>Chatbot:</strong> User interface</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-color" style="background: #cc99ff;"></div>
|
||||
<span><strong>Audit:</strong> Compliance trail</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Data Flow Stages</h2>
|
||||
<div class="flow-step">
|
||||
<span>1. Source</span>
|
||||
<p>SAP ERP systems extract quotation, discount, and customer data daily at 05:00 UTC. Data includes quotation identifiers, line items, pricing, and discount conditions.</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>2. Raw</span>
|
||||
<p>Snowflake External Stage ingests CSV/Parquet. Raw schema preserves source structure without modification. Data quality checks validate row counts, null patterns.</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>3. Staging</span>
|
||||
<p><code>stg_*</code> models rename columns, cast data types, and handle nulls. Example: source quotation ID → <code>quote_id</code>, source amount → <code>net_value</code>. No business logic yet.</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>4. Transform</span>
|
||||
<p><code>trf_quotation</code> joins customer policies, product portfolios, and agreement discounts. Calculates discount totals, applies governance flags (e.g., <code>discount_above_ceiling</code>). Two models: <code>trf_quotation</code> (BI) and <code>trf_chatbot_quotation</code> (LLM-optimized).</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>5. Semantic</span>
|
||||
<p>Snowflake DDL semantic views expose curated dimensions and facts. Includes synonyms for natural language (e.g., "desconto cliente" → <code>discount_customer_pct</code>). Single source of truth for BI dashboards and LLM queries.</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>6. Cortex</span>
|
||||
<p><a href="https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-analyst" target="_blank" rel="noopener noreferrer">Cortex Analyst</a> applies guardrails: hides raw discount percentages, protects customer names, checks data freshness, enforces role-based access, and logs all access. LLM queries go through here.</p>
|
||||
</div>
|
||||
<div class="flow-step">
|
||||
<span>7. Consume</span>
|
||||
<p><strong>Snowflake UI:</strong> Dashboards for ops/sales teams (full data visibility). <strong>Internal Workspace:</strong> Chatbot interface (guardrail-filtered data). Both consume the same semantic layer; governance rules ensure consistency.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Why This Architecture?</h2>
|
||||
<p><strong>Governance-first:</strong> Guardrails (discount policies, protection of personal and customer information, and audit logs) are embedded in the data layer, not bolted on to the LLM prompt. The bot can't bypass policy; it's enforced by Cortex + dbt.</p>
|
||||
<p><strong>Semantic contract:</strong> The semantic layer (sv_quotation, sv_chatbot_quotation) defines the agreement between data engineers, BI teams, and LLM consumers. Column definitions, synonyms, and access rules are version-controlled.</p>
|
||||
<p><strong>Team efficiency:</strong> dbt lineage shows how every dimension flows from SAP → semantic layer. Data engineers and business analysts can trace a discount calculation back to SAP in seconds. Onboarding takes days, not weeks.</p>
|
||||
<p><strong>Prototyping speed:</strong> Snowflake + Cortex means we didn't build custom infrastructure. POC shipped in weeks. Cortex guardrails were ready on day 1; no custom policy engine to build.</p>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Expected Outcomes</h2>
|
||||
<p><strong>Latency:</strong> Quote research from 5-30 minutes (email + manual lookup) → <1 minute (chatbot query)</p>
|
||||
<p><strong>Scalability:</strong> 30+ salespeople can query the chatbot concurrently. Snowflake handles scale; dbt models are stateless (scale linearly).</p>
|
||||
<p><strong>Governance:</strong> All discount exceptions flagged and audited. Policy violations are visible and traceable.</p>
|
||||
<p><strong>Team capability:</strong> Once-daily batch + semantic layer pattern is reusable. Next use cases (contracts, pricing, serviceability) ship faster by leveraging the same playbook.</p>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Learn More</h2>
|
||||
<p>
|
||||
See <code>README.md</code> for full narrative, decisions, and lessons learned.
|
||||
Check <code>dbt/</code> for anonymized SQL models.
|
||||
Review <code>cortex/cortex_governance_config.yaml</code> for guardrails and access control.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user