Module 2: Databricks Workspace & Environment

• The sidebar changes between releases. Look for names, not positions

• Workspace holds notebooks and workspace files (modules, configs, small text)
• Your files: /Workspace/Users/<your-email>/
• On serverless the working directory isn't guaranteed: use absolute paths

# helpers.py
def add_vat(price, rate=0.21):
    return round(price * (1 + rate), 2)

import sys

user = spark.sql("SELECT current_user()").first()[0]
sys.path.append(f"/Workspace/Users/{user}/dbx-course")

from helpers import add_vat
print(add_vat(100))

121.0

• Cells run in the notebook's default language unless they start with a magic like %sql
• Free Edition: Python and SQL only (no R, Scala)
• Each language has its own REPL: a Python variable is invisible to SQL

spark.range(3).createOrReplaceTempView("three_rows")

%sql
SELECT * FROM three_rows

display(_sqldf)    # result of the last SQL cell

• Shift+Enter runs the cell and moves on
• Cmd/Ctrl+Shift+F formats code
• Autocomplete needs attached compute. Python notebooks have a debugger

• Types: text, dropdown, combobox, multiselect
• Arguments: name, default, choices (not for text), label
• Job parameters arrive as widget values (Module 10)

dbutils.widgets.text("country", "NL", "Country code")
dbutils.widgets.dropdown("min_qty", "1", ["1", "2", "3", "4", "5"], "Minimum quantity")

country = dbutils.widgets.get("country")
min_qty = int(dbutils.widgets.get("min_qty"))
print(country, min_qty)

%sql
SELECT :country AS country, CAST(:min_qty AS INT) AS min_qty

• Widget as table or column name: IDENTIFIER(:table_name)
• Clean up: dbutils.widgets.removeAll()

• Git folder = workspace folder linked to a remote repo
• Needs a Git account (e.g. GitHub) and credentials configured first

• Merge, rebase, reset are in the dialog menu (rebase and reset rewrite history)

• Runtime = software compute runs: a Spark version, Python and system libraries, Databricks optimisations
• LTS = long-term support, about three years
• Every runtime has an ML variant with ML libraries preinstalled
• Serverless: no runtime choice. You pick an environment version instead

From the release notes as of October 2026. Check the current list.

print(spark.version)

• Serverless jobs don't support every workload. Streaming: continuous scheduling only with bounded triggers like Trigger.AvailableNow
• Pricing follows the compute that runs the task (Module 18)

%pip install humanize==4.9.0

dbutils.library.restartPython()    # variables before this are lost

import humanize
print(humanize.intcomma(1234567))

1,234,567

• %pip is notebook-scoped and gone next session: put it at the top
• It doesn't restart Python for you
• Other sources: wheel in a volume (/Volumes/.../pkg.whl), workspace wheel, Git URL
• Reproducible serverless setup: Environment side panel with pinned versions
• Environment version 5+: %uv pip for faster installs while iterating (not for reproducible environments)

• External locations: existing cloud storage governed by Unity Catalog
• A volume belongs to a schema: create the schema and volume first
• Module 4 reuses this schema

catalog = spark.sql("SELECT current_catalog()").first()[0]
schema = "dbx_course"

spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")
spark.sql(f"CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.raw")

path = f"/Volumes/{catalog}/{schema}/raw"
with open(f"{path}/hello.csv", "w") as f:
    f.write("id,name\n1,Ada\n2,Grace\n")

spark.read.option("header", True).csv(f"{path}/hello.csv").show()

+---+-----+
| id| name|
+---+-----+
|  1|  Ada|
|  2|Grace|
+---+-----+

display(dbutils.fs.ls(path))