Module 19: Advanced Lakehouse Architecture

• Silver reads from bronze, often as a stream: don't write straight from ingestion to silver
• Medallion is a recommended pattern, not a requirement: keep it simple for small tables
• At scale add: one schema per layer and domain, naming rules, owners, and quality rules at each hop
• Ingestion frequency is a cost and latency choice: continuous = fastest and dearest, triggered = balanced, batch = cheapest

%sql
-- one schema per layer, per domain: a naming convention you can read
SHOW SCHEMAS

• Unified engine: same DataFrame code for batch and streaming
• Decide per dataset, not per company. Most tables don't need seconds
• Triggered pipelines give near-real-time at lower cost than always-on

• Lambda: exact but slow batch + fast but approximate stream. The cost: two implementations that must agree
• Kappa: stream only. Reprocessing needs a replayable log (such as Kafka with long retention)
• Delta tables are both a streaming sink and source, and a batch table: one code path with different triggers
• Reprocess by re-reading bronze in batch, or by a one-time backfill flow (Module 9.7)
• Keep bronze long enough to replay: retention is a design decision

• Land the raw change events in bronze, append-only: this is your audit trail and replay source
• Apply changes to silver with AUTO CDC (Module 9.9) or MERGE (Module 5.8) with a sequence column to order events
• Choose SCD type 1 (current state) or type 2 (history) per table (Module 8.10)
• Deletes: a soft delete flag or a real delete (APPLY AS DELETE), per policy and GDPR
• Managed ingestion connectors (Lakeflow Connect) can capture from databases and SaaS apps: check availability for your workspace
• Test out-of-order, duplicate and late events (Module 14.9)

• Data as a product: each product has an owner, a documented schema, a quality target, a freshness SLA and a place to ask questions
• Domains publish gold-level tables, views or metric views. Consumers don't read a domain's raw layers
• Unity Catalog supplies: catalogs per domain, tags (such as certified), comments, lineage, access requests, and sharing
• The platform team provides the paved road: templates, bundles, policies, CI/CD
• Federated governance: global rules, domain-level ownership

• Unity Catalog is the shared layer: users, groups and data policies at the account level
• Use workspace-catalog binding so prod data isn't visible from dev workspaces
• Deploy to all workspaces from the same repo with different targets (Module 15)
• Share across metastores or regions through sharing, not by pointing at storage
• More workspaces = more to administer. Don't split without a reason

• Reasons: a regulation, an acquisition, a customer requirement, or resilience
• Costs: more skills, more networking, egress fees, two sets of security setups

• The platform is the same on each cloud: Unity Catalog, jobs, SQL, notebooks. Storage and networking differ
• Keep compute near data: cross-cloud reads are slow and cost egress
• Define one governance model (naming, tags, groups) and apply it everywhere with code
• Avoid cross-cloud designs unless a requirement forces them

• Model business entities once, as conformed dimensions (customer, product, date) used by many facts
• Document grain, keys and definitions. A metric view keeps measures (revenue) in one place
• Primary and foreign keys in Unity Catalog are informational: they document and can help the optimizer, but aren't enforced
• Don't over-model: start from the questions the business asks

%sql
-- document the model where people will see it
COMMENT ON TABLE orders IS 'Order header. Grain: one row per order. Owner: sales-data'

%sql
-- needs a Unity Catalog table; features and runtime versions apply
CREATE OR REPLACE TABLE orders_iceberg_readable (order_id INT, amount DOUBLE)
TBLPROPERTIES (
  'delta.columnMapping.mode' = 'id',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg')

• Iceberg reads (formerly UniForm): Delta tables expose Iceberg metadata. Clients get read-only access
• Requirements: Unity Catalog table, column mapping, no deletion vectors (purge with REORG first)
• Metadata generation is asynchronous after each commit
• Unity Catalog can act as an Iceberg REST catalog for external engines (see the current docs)
• Prefer open formats and interfaces: it keeps your options open

• Sharing gives live, read-only access without copying to the recipient
• Share gold or views, not raw tables. Filter rows and columns in a view
• Recipient costs: their compute. Provider costs: egress across regions or clouds, if any
• Governance doesn't cross boundaries: recipients apply their own access control and lineage
• You can revoke access any time

• Databricks recommends managed disaster recovery for cross-region DR where available (AWS and Azure). DIY covers what it doesn't
• DR doesn't protect against corruption: bad data replicates too. Use time travel and backups for that
• Test failover regularly: shut down primary, start secondary, repoint BI and clients, and measure
• Plan failback as a maintenance window

%sql
-- deep clone: a full copy you could replicate
CREATE OR REPLACE TABLE orders_dr_copy DEEP CLONE orders

Guiding principles (from the docs):

• Curate data and offer trusted data as products
• Eliminate silos and minimise data movement
• Democratise value creation through self-service
• Adopt organisation-wide governance for data and AI
• Encourage open interfaces and open formats

• Build to scale, optimising for performance and cost

• Use the pillars as a review checklist at design time and after incidents

• Write down trade-offs you accept (for example latency vs cost)
• Module 22 turns these into a production readiness checklist