SELECT * FROM samples.nyctaxi.trips; -- catalog.schema.table
Module 1: Introduction to Databricks
What Databricks is, how its architecture splits across planes, and how to get a free workspace running.
The vocabulary around Spark: what the platform is, where code runs, how it differs from a warehouse, a lake and open-source Spark.
- Topics 1 to 8: short explanations
- Topic 9: your first hands-on step (free workspace + first cell)
- Each topic starts with By the end. Tick Mark this topic as done to track progress in your browser
1.01Concept
What is Databricks?
By the end: explain Databricks to a teammate in two sentences and name the main things it adds on top of Spark.
- Cloud platform for data engineering, analytics and AI
- Core idea, the lakehouse: data stored once, in open formats on cloud storage. SQL, Python, streaming and ML all work on it under one set of access rules
- You already know the engine (Spark). Databricks adds everything around it
| Layer | What it adds | Module |
|---|---|---|
| Compute | Managed clusters or serverless | 2, 12, 18 |
| Workspace | Notebooks, files, Git folders, dashboards | 2 |
| Storage format | Delta Lake: transactions, time travel | 5 |
| Governance | Unity Catalog | 6 |
| Pipelines | Lakeflow: ingest, declarative pipelines, jobs | 7, 9, 10 |
| Analytics | SQL warehouses, AI/BI dashboards, Genie | 13 |
| AI and ML | MLflow, serving, vector search, agents | 20 |
Remember
Lakehouse = lake storage costs and flexibility + warehouse reliability and governance.
1.02Concept
Data Warehouse vs Data Lake vs Data Lakehouse
By the end: choose between a warehouse, a lake and a lakehouse for a given workload and say why.
- Warehouse: fast, trustworthy SQL for analysts
- Lake: cheap storage for everything, no guarantees
- Lakehouse: a table layer with transactions on top of lake storage
- The docs name three pieces: Spark (compute), Delta Lake (storage layer), Unity Catalog (governance)
1.03Concept
Databricks Lakehouse Platform
By the end: name the layers of the Databricks platform and the product that belongs to each.
Unity CatalogPermissions, lineage, auditing, discovery
Engines and toolsNotebooks, Lakeflow, SQL warehouses, dashboards, MLflow
Delta LakeOpen table format: ACID, schema enforcement, time travel
Cloud object storageParquet files in your cloud or default storage
| Product | For |
|---|---|
| Lakeflow Connect | Ingesting from files, databases, SaaS |
| Lakeflow Spark Declarative Pipelines | Declare tables, platform runs them (was Delta Live Tables) |
| Lakeflow Jobs | Schedule notebooks, scripts, pipelines |
| Databricks SQL | SQL analytics and BI |
| AI/BI dashboards, Genie | Dashboards, natural-language questions |
| MLflow, Mosaic AI | Experiments, serving, agents |
| Databricks Apps | Python web apps next to your data |
| Lakebase | Managed Postgres |
Names change
Delta Live Tables is now Lakeflow Spark Declarative Pipelines. Databricks Asset Bundles are now Declarative Automation Bundles. Old blog posts use the old names.
1.04Concept
Databricks vs Apache Spark
By the end: tell which parts of your PySpark code move to Databricks unchanged and which need attention.
| Open-source Spark | Databricks | |
|---|---|---|
| Cluster | You provision and tune | Managed or serverless |
| Tables | Files + a metastore you run | Delta + Unity Catalog |
| Notebooks | Bring your own | Built in, Git integration |
| Scheduling | cron, Airflow | Lakeflow Jobs |
| Session | You create it | spark already exists |
- Each Databricks Runtime ships a specific Spark version
- Seen in action in Modules 2 and 3
1.05Concept
Databricks vs Snowflake
By the end: compare the two platforms along the dimensions that matter for a decision, without relying on marketing claims.
| Databricks | Snowflake | |
|---|---|---|
| Starts from | Spark, lakehouse: engineering, streaming, ML | Cloud warehouse: SQL analytics |
| Interfaces | Notebooks, SQL, Python, Scala | SQL first, others added |
| Data format | Delta and Iceberg on open files | Own managed storage, growing open-format support |
| Strong fit | Heavy engineering, streaming, ML/AI | SQL analytics, BI, simple admin |
| Pricing | Consumption (DBUs + cloud cost) | Consumption (credits + storage) |
Starting point only
The Databricks column comes from Databricks docs. The Snowflake column is general background I did not verify against Snowflake's docs. Check both vendors before deciding.
- How to decide: 1) workload, 2) team skills and tools, 3) governance and cloud needs, 4) small proof of concept on each
- Many organisations use both
1.06Concept
Databricks Architecture
By the end: name the main objects in a Databricks account and say where each one lives.
- Account: top level. Identity, workspaces, metastores, billing
- Workspace: where people run ingestion, exploration, jobs, ML
- Workspace file system data: notebooks, queries, dashboards, Git folders
- Default storage: managed storage for serverless workspaces
- DBFS (
dbfs:/): older file system, no longer recommended, may be disabled
On Free Edition
One workspace, one metastore, no account console. Account-level topics are explanation only.
1.07Concept
Control Plane vs Compute Plane
By the end: draw the control plane and the compute plane from memory and say which one your data is processed in.
- Control plane: Databricks-managed backend and web app, in Databricks' account
- Classic compute plane: clusters in your cloud account and network
- Serverless compute plane: in Databricks' account, same region, isolated per workspace
- Classic workspaces also have a storage bucket in your account. Serverless uses default storage
Why it matters
Security, networking and cost questions start with "which plane?". "The cluster is in our VPC" means classic.
1.08Concept
Databricks on Azure, AWS, and GCP
By the end: say what differs between running Databricks on AWS, Azure and Google Cloud.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
| Storage | S3 | ADLS Gen2 | Cloud Storage |
| Identity | IAM roles | Microsoft Entra ID | Google identity |
| Docs | docs.databricks.com | Microsoft Learn | docs.databricks.com |
- Same platform everywhere. Delta, Unity Catalog, Spark and Jobs carry across clouds
- Docs have a cloud switcher: read the page for your cloud
- Azure Databricks is an Azure service (billing, support, identity)
On Free Edition
You don't pick a cloud or connect storage. You get serverless compute and default storage.
1.09Hands-on with caveats
Databricks Free Edition and Workspace Setup
By the end: have a working Free Edition workspace, run your first notebook cell, and know the limits you will work within.
Run it in Databricks
- Open the Databricks Free Edition page in the docs and choose Sign up (email, Google or Microsoft)
- In the sidebar: New, then Notebook
- Paste the cell below and press Shift+Enter
display(spark.sql("""
SELECT current_catalog() AS catalog,
current_schema() AS schema,
current_user() AS user
"""))
display(spark.read.table("samples.nyctaxi.trips").limit(10))
No samples catalog?
I couldn't confirm it exists in every Free Edition workspace. Check Catalog in the sidebar. If missing, skip the second cell. Module 4 generates its own data.
| Area | Free Edition limit |
|---|---|
| Compute | Serverless only, no custom config, trusted outbound domains only |
| SQL warehouse | One, 2X-Small |
| Jobs | 5 concurrent tasks |
| Pipelines | One active per type |
| Apps | Up to 3, each stops after 24 h (restartable) |
| Lakebase | One project |
| Not supported | R, Scala, custom storage locations, online tables, clean rooms |
| Admin | One workspace and metastore, no account console, SSO, SCIM |
| Use | Non-commercial, no SLA, inactive accounts may be deleted |
- Over quota: compute pauses until the limit resets. Nothing is deleted
- Community Edition was retired in 2025. Free Edition replaces it
Stay in quota
Use LIMIT while exploring, avoid big generated data, stop cells you no longer need.
Recap
What you can do now
- Lakehouse vs warehouse vs lake
- Platform layers and Lakeflow products
- What changes for PySpark on Databricks
- Control plane, serverless and classic compute planes
- A working Free Edition workspace and your first cell