DatabricksZero to Advanced
/
Course/Level 7 · Quick Reference & Revision

Module 25: Visual Revision & Mental Models

Diagrams and decision trees for revision.

12 concept12 topics
Topics in this module
  1. Databricks Architecture Mind Map
  2. Spark Execution Lifecycle
  3. DataFrame Transformation Flow
  4. Delta Lake Transaction Lifecycle
  5. Medallion Architecture Diagram
  6. Batch vs Streaming Decision Tree
  7. Ingestion Strategy Decision Tree
  8. Join Strategy Decision Tree
  9. Delta Optimization Decision Tree
  10. Pipeline Failure Recovery Flowchart
  11. Production Deployment Lifecycle
  12. End-to-End Data Engineering Architecture

One picture per idea. Use this module to revise: look at the picture, say it out loud, then check the notes.

  • Diagrams recap earlier modules, each topic names where it was taught
  • Decision trees are written as question → answer so you can run through them quickly
  • Try drawing each one from memory on paper. Gaps show what to reread
25.01Concept

Databricks Architecture Mind Map

By the end: see the whole platform on one page.

DataDelta tablesVolumesUnity CatalogComputeServerless notebooksSQL warehouseJobs computeBuildPySparkSQLLakeflow pipelinesRunLakeflow JobsTriggersRepairGovernPermissionsLineageAudit
Five groups cover most of the platform.
  • Data lives in Delta tables and volumes, governed by Unity Catalog
  • Compute is what runs code. On Free Edition it is serverless
  • Build with PySpark, SQL or declarative pipelines. Run with jobs
  • Govern across all of it: one set of permissions
  • Taught in Modules 1 to 3
25.02Concept

Spark Execution Lifecycle

By the end: follow a Spark query from code to result.

Your codeLogical planOptimized planTasks on compute
Lazy until an action asks for the result.
  • Transformations only build the plan. An action triggers the run
  • Optimizer reorders and prunes: push filters down, read fewer columns
  • A shuffle splits the work into stages. Each stage is many parallel tasks
  • Look at it: df.explain(), then the query profile
  • Taught in Modules 5 and 12
25.03Concept

DataFrame Transformation Flow

By the end: read a DataFrame pipeline as a chain of steps.

ReadFilter / selectJoinAggregateWrite
Narrow steps stay local. Join and aggregate usually shuffle.
Step Cost
Read Scans files. Less data is cheaper
Filter / select Free: done in place
Join, group, sort Moves data between tasks
Write The action that runs it all
  • Reduce rows before the wide steps
  • Taught in Module 5
25.04Concept

Delta Lake Transaction Lifecycle

By the end: see what happens when you write to a Delta table.

Write filesAdd log entryNew versionReaders see it
The log is the table. Data files are only referenced by it.
  • A write adds new Parquet files, then commits one log entry. Readers see all of it or none
  • Update and delete rewrite files, they do not edit in place
  • Two writers touching the same files: one commit fails and retries or errors
  • DESCRIBE HISTORY shows versions; VERSION AS OF reads an old one
  • VACUUM removes old files, which ends time travel past the retention period
  • Taught in Module 6
25.05Concept

Medallion Architecture Diagram

By the end: draw the medallion layers and what each one promises.

SourcesBronzeraw, append-onlySilverclean, dedupedGoldbusiness tables
Each layer adds trust.
Layer Promise
Bronze Nothing lost: raw plus load metadata
Silver Valid, typed, deduplicated; bad rows quarantined
Gold Shaped for questions: facts, dimensions, aggregates
  • Rebuild silver and gold from bronze when logic changes
  • Taught in Modules 7 and 8
25.06Concept

Batch vs Streaming Decision Tree

By the end: decide between batch and streaming.

Need fresh data in minutes?Yes: streaming or small batchesNo: scheduled batch
Question If yes If no
Is freshness needed in seconds? Streaming, always on (not on Free Edition) Next question
Needed in minutes to hours? availableNow on a schedule Daily batch
Is the source an event stream? Streaming reads File or table reads
Is the logic stateful (windows, dedup)? Streaming with watermarks Plain batch SQL
  • Streaming costs more to run and to operate. Use it only when the business needs it
  • On serverless, streaming runs as availableNow batches
  • Taught in Module 11
25.07Concept

Ingestion Strategy Decision Tree

By the end: choose how to ingest data.

Source Choose
Files arriving in cloud storage or a volume Auto Loader (or a pipeline streaming table)
One-off or small files COPY INTO, or spark.read
Database changes A managed connector or CDC feed, then AUTO CDC
Event stream (Kafka and similar) Structured Streaming read
REST API Python job with watermark and retry, raw JSON first
SaaS apps Managed connectors
Data from another Databricks workspace or company Delta Sharing
Are the sources files?Yes: Auto LoaderNo: see the table above
  • Always land raw data first, then transform
  • Taught in Modules 7, 9, 11 and 21
25.08Concept

Join Strategy Decision Tree

By the end: pick a join strategy by table size.

Is one side small?Yes: broadcast joinNo: sort-merge, check skew
Situation Action
Small dimension + big fact broadcast(dim)
Both big, even keys Let Spark choose; make sure filters come first
Both big, a few huge keys Salt the key or rely on adaptive skew handling
Need only existence left_semi / left_anti
  • Compare row counts before and after a join: a jump means a bad key
  • Taught in Module 12
25.09Concept

Delta Optimization Decision Tree

By the end: pick the right Delta optimization.

Symptom Action
Many small files OPTIMIZE (or auto compaction / predictive optimization)
Queries filter on a few columns Liquid clustering on those columns
Table too large for its use Retention: delete old data, VACUUM
Slow MERGE Match on the key, filter the target, cluster by the key
Read slow after many updates OPTIMIZE, check deletion vectors
Stale statistics ANALYZE TABLE ... COMPUTE STATISTICS
Slow Delta query?Layout: OPTIMIZE, clusterLogic: read query profile
  • Check file count (DESCRIBE DETAIL) before reaching for tuning
  • Taught in Module 12
25.10Concept

Pipeline Failure Recovery Flowchart

By the end: recover from a failed run in the right order.

Run failedFind the failing taskFix the causedata, code, permissionRepair runfailed and downstream tasksCheck row counts
Cause Recovery
Transient (timeout, network) Retry; repair the run
Bad input data Quarantine, fix, rerun the day
Code bug Fix, redeploy, then repair or backfill
Partial write The load must be idempotent so the rerun is safe
  • Idempotent loads make recovery boring
  • Taught in Modules 10, 16 and 21
25.11Concept

Production Deployment Lifecycle

By the end: follow code from your laptop to production.

BranchPR + testsMergeDeployRun in prod
Nothing reaches prod without passing tests.
  • Develop and test in a dev target. Prod is deployed only from main or a tag
  • The deploy tool is a bundle or the Jobs API; the run identity in prod is a service principal
  • Rollback = deploy the previous tag
  • Taught in Modules 15, 26, 27 and 28
25.12Concept

End-to-End Data Engineering Architecture

By the end: draw an end-to-end data engineering architecture from memory.

SourcesFilesDatabaseAPIIngestAuto LoaderConnectorsPython jobsProcessBronzeSilverGoldServeSQLDashboardsMLOperateJobsMonitoringCI/CD
Governance (Unity Catalog) spans every column.
  • Left to right: sources → ingest → process → serve
  • Across the bottom: governance, orchestration, monitoring, cost
  • In an interview, start with this picture and walk through it, one column at a time
  • Taught in Modules 19 and 23
Recap

What you can do now

  • Twelve pictures: platform, Spark plan, DataFrame flow, Delta write, medallion
  • Five decisions: batch or stream, ingestion, join, Delta optimization, recovery
  • Two lifecycles: deployment and the end-to-end architecture