Course/Best practices

Best practices

One page of checklists, kept apart from the lessons.

Areas

Checklists by area. Short on purpose: each line is a rule you can check your own project against.

Workspace and compute

Develop on interactive compute (serverless on Free Edition). Run scheduled work as jobs
Keep reusable code in modules or wheels, not %run
Number and name notebooks in reading order (01-ingest, 02-clean)
Pin library versions (==x.y.z) and keep dependencies in the environment, not ad hoc installs
Parameterize with widgets or job parameters. No hard-coded paths, catalogs or user names
Don't rely on caching on serverless. Write expensive intermediates to Delta tables
Know which compute plane a workload runs in before a security or cost review
Use LIMIT while exploring, especially inside the Free Edition quota

Built-in functions first, pandas_udf next, plain Python UDFs last
Filter early and select only the columns you need
Shuffle as little as you can. Look for Exchange in explain()
Define explicit schemas for CSV and JSON. No inferSchema in pipelines
Wrap every filter condition in parentheses. Test nulls with isNull(), never == None
Compare row counts before and after joins. Dedupe the right side first
Bring only small results to the driver. No collect() or toPandas() on big data
Pass values into SQL with parameter markers, not f-strings
Don't call withColumn in a long loop: use withColumns or one select
Handle divide-by-zero explicitly (try_divide): serverless runs with ANSI mode on
Don't repartition without a measured reason

Delta tables, not loose CSV or Parquet, as the default format
Land raw data as-is first, clean it next, model it last (bronze, silver, gold)
Use MERGE for row-level changes. overwrite replaces the whole table
Avoid many tiny files, and avoid coalesce(1) on large data
Prefer the built-in layout features (liquid clustering, predictive optimization) over hand-built partitioning. Check the docs for your workspace
Change schemas deliberately: use overwriteSchema only when you mean it

Code lives in src/ as a package. Notebooks stay thin and call functions
Transformations are DataFrame-in, DataFrame-out functions
One config per environment, injected at run time
Unit-test the functions without a cluster where you can. Keep a few integration tests for Spark behaviour
Lint and format automatically (pre-commit and CI)
Build a versioned wheel and deploy the same artifact to every environment. Bump the version on every release
Don't list pyspark as a dependency: the runtime provides it
Read parameters in one place (parse_args) and fail early on a bad value
Log with logging. Fail loudly with a clear error, not silently

Anything scheduled runs as a job, not a notebook you start by hand
Serverless jobs compute is the default choice where it supports the task
Make tasks idempotent so a rerun or repair run is safe
Set retries, timeouts and failure notifications on every job
Pass parameters. Don't keep per-environment copies of a notebook
Define jobs as code (Declarative Automation Bundles) and deploy them from CI
Run production jobs as a service principal
Retry only transient failures. Don't retry a data quality check
Reference job parameters in tasks ({{job.parameters.run_date}}) so one value reaches every task

Airflow orders and schedules. Databricks does the Spark work. Don't move data through Airflow
Keep the job definition in one place (Databricks or Airflow), not both
Authenticate with a service principal. Keep credentials in an Airflow connection or secrets backend
Make tasks idempotent, and set retries in one layer so they don't multiply
Pass the run date as a job parameter. Don't compute it inside the notebook
Alert from Airflow, and log the Databricks run URL so failures are one click away
Keep DAG files light: no heavy code at import time
When Airflow triggers the job, remove the Databricks schedule so it doesn't run twice
Call the job by name from a variable, not by a hard-coded job id