Databricks, from your first notebook to production
28 modules take you from the workspace and Delta tables to pipelines, testing, CI/CD and architecture. You build everything on Databricks Free Edition, so the only cost is your time.
How each topic works
Every topic opens with what you will be able to do, explains the idea briefly, and then has you run it.
Read, then run
Code appears as notebook cells with a copy button. Paste each into a cell in your own workspace and press Shift+Enter.
Labels set expectations
Hands-on topics run on Free Edition. Some need a Git account or a paid workspace. Others are concept-only because the free tier does not offer the feature.
Progress stays with you
Tick a topic when you finish it. Ticks are stored in this browser only, so they will not follow you to another device.
Before you start
You need a desktop browser and an email, Google or Microsoft account for the free sign-up. You should be comfortable with Python functions and with DataFrame basics such as select, filter and groupBy. Module 1 walks you through creating the workspace.
Want the project path first?
Levels 8 and 9 build one small project and run it two ways. Take this route. They assume you already know Python, PySpark, Git and Airflow.
- Modules 1–4 Databricks and the workspace, Spark, working with data
- Modules 5–6 Delta tables and Unity Catalog
- Module 26 Python project on Databricks
- Module 27 Run it with Databricks Jobs
- Module 28 Run it with Airflow
- Best practices One separate checklist page
Roadmap
9 levels. The bars next to a level show how demanding it is.
Level 1 · Foundations
BeginnerIntroduction to Databricks
What Databricks is, how its architecture splits across planes, and how to get a free workspace running.
Databricks Workspace & Environment
Find your way around the workspace, notebooks, compute and file storage.
Apache Spark Fundamentals
The Spark ideas you need to read plans, reason about performance and avoid surprises on serverless.
Working with Data in Databricks
Read, shape, join and write data with PySpark and SQL on a practice dataset you generate yourself.
Level 2 · Core Data Engineering
IntermediateDelta Lake Fundamentals
Delta tables: transactions, MERGE, time travel, maintenance and liquid clustering.
Unity Catalog & Data Governance
Catalogs, schemas, volumes, privileges, lineage and sharing with Unity Catalog.
Data Ingestion
Bring data in with COPY INTO, Auto Loader, JDBC, APIs and change data capture.
Data Transformation & Modeling
Bronze, silver and gold layers, dimensional modeling and slowly changing dimensions.
Lakeflow Pipelines
Declarative pipelines with streaming tables, materialized views and expectations.
Workflow Orchestration
Schedule, chain and monitor work with Lakeflow Jobs.
Level 3 · Performance & Production Engineering
AdvancedStructured Streaming
Structured Streaming: triggers, checkpoints, state, watermarks and Kafka.
Spark & Databricks Performance Optimization
Read query plans, tune joins and shuffles, and understand AQE and Photon.
Databricks SQL & Analytics
SQL warehouses, advanced SQL, AI/BI dashboards and Genie.
Testing & Data Quality Engineering
Test PySpark code and pipelines, and enforce data quality.
CI/CD & Deployment
The CLI, bundles, local development and CI/CD.
Production Monitoring & Operations
Monitor, alert, troubleshoot and operate pipelines in production.
Level 4 · Architecture & Enterprise Adoption
ExpertSecurity & Enterprise Governance
Identity, secrets, network security and enterprise governance.
Cost & Resource Optimization
DBUs, compute choices, cluster policies and FinOps.
Advanced Lakehouse Architecture
Lakehouse architecture at scale: mesh, multi-workspace, DR and open formats.
Databricks AI & Machine Learning
MLflow, model serving, vector search, RAG and agents.
Level 5 · Industry Standards, Principles & Design Patterns
Level 6 · Hands-On Industry Projects
Level 7 · Quick Reference & Revision
Level 8 · Small Project: Build It and Run It with Databricks Jobs
Python Project on Databricks
Structure the small project for Databricks: layout, config, entry points, tests, wheel and Git folders. Assumes you know Python and Git.
Run the Project with Databricks Jobs
Run the project as a Databricks job: tasks, parameters, schedule, retries, repair runs, job as code, CI deploy.