Module 10: Workflow Orchestration

• Job = tasks + dependencies + trigger + parameters + notifications
• Task = one unit of work: notebook, Python script, wheel, SQL, pipeline, dbt, and more
• Trigger = what starts the job: a schedule, a file, a table update, or manual
• Jobs orchestrate. A pipeline (Module 9) is something a job can run as one task

Create it:

1. Jobs & Pipelines → Create → Job
2. Task name ingest. Type Notebook. Pick the ingest notebook
3. Compute: Serverless (the only option on Free Edition)
4. Run now, then open the run

• Each run has an ID, a state and a duration. A task's output and logs are one click away
• Name tasks for what they do (ingest, transform): names are used in references like {{tasks.ingest.values.row_count}}
• Develop in the notebook, test interactively, then run it as a job. Behaviour should match

Add tasks: Add task, then Depends on → pick the upstream task.

• The tasks form a DAG: directed, no cycles
• Tasks with no link between them run in parallel (within the compute and the 5-task cap)
• Write tasks so each one can run again safely: MERGE and overwrite, not blind appends

Add a trigger: Job details → Add trigger.

• Cron uses Quartz syntax: 0 0 6 * * ? = every day at 06:00
• Pick a time zone. In a daylight-saving zone, hourly jobs can skip or double at the change. Use UTC for strict hourly timing
• Scheduling is not low-latency: runs can start a few minutes late
• Pausing a trigger stops new runs. Active runs finish
• Default max concurrent runs is 1: a run is skipped (or queued, if queueing is on) while another is active

# Python script task: read arguments from the command line
import sys
run_date = sys.argv[1]

• Notebook tasks are quickest to start. Script and wheel tasks are better for tested, versioned code (Module 26)
• Source can be the workspace or a Git repository and branch/tag (Module 15, 26)
• Dependencies for serverless tasks go in the task's environment
• A notebook run as a job needs defaults only for interactive use: the default can hide a missing job parameter

• Add task → Pipeline → choose your pipeline (Module 9)
• A pipeline task starts an update and waits for it. It succeeds or fails with the update
• Put it between other tasks: ingest files → pipeline → report

• Pass values with the task's Parameters box using dynamic references (next topic)
• Full refresh is an option on the task. Leave it off for routine runs
• Job-run pipelines are retried and restarted on recoverable errors, unlike interactive UI runs
• Pipeline tasks are not parameterised per backfill slice: for history use a one-time append flow (Module 9.7)

Job parameters, set under Job details → Parameters:

Reference them in a task's parameters:

Task values: set in one task, read by another:

# in ingest
dbutils.jobs.taskValues.set(key="row_count", value=n)

• Give the transform task the parameter rows = {{tasks.ingest.values.row_count}}, then read it with dbutils.widgets.get("rows")
• Databricks recommends the reference syntax over taskValues.get: renaming a task won't break it
• A task value must be JSON and at most 48 KiB
• Parameters are strings in widgets: cast them
• A job parameter wins over a task parameter with the same key

If/else task: runs one branch based on a condition:

1. Add task → If/else condition named has_rows
2. Condition: {{tasks.ingest.values.row_count}} > 0
3. Set transform to depend on has_rows (true)

• Try it: today's date has no orders, so the true branch is skipped. Run with run_date = a 2025 date and it runs
• Operators: ==, !=, >, >=, <, <=
• == and != compare strings (12.0 == 12 is false). The others compare numbers
• Values must be number, string or boolean. A boolean true becomes the string "true"

Run if (set per task) reacts to upstream status:

• Use At least one failed or All done for cleanup and alert tasks
• A skipped (Excluded) upstream counts as success for run-if

Per task: Retries (how many, wait between attempts, retry on timeout) and Timeout.

Per job: Timeout under run duration, an expected-duration warning, and Notifications.

• Notify by email. Slack, Teams, PagerDuty and webhooks need an admin-set destination
• Job-level notifications are not sent while a failed task is retrying. Use task-level ones for every failure, or mute until the last retry
• Set max concurrent runs deliberately. Extra runs are skipped, or queued (up to 48 hours) when queueing is on

• Several tasks that depend on the same upstream task start together
• A downstream task with several dependencies waits for all (default) or follows its Run if rule
• For each repeats a task over a list:

[{"table": "orders"}, {"table": "customers"}, {"table": "products"}]

• Nested task parameter: {{input.table}}. The list can come from a task value or a job parameter
• Concurrency controls how many iterations run at once (default 1)
• A For each cannot contain another For each
• Free Edition limit: 5 concurrent tasks in the account. More wait

• Jobs & Pipelines → Runs: all recent runs, filter by status, job, time
• Matrix view on a job: one column per run, one cell per task. Green success, red failure, pink skipped, yellow waiting to retry
• Run details: Graph, Timeline and List views. Click a task for output, logs, code and metrics
• The Timeline shows slow tasks and where branches overlap
• A run's result depends on its leaf tasks (those with nothing after them)

-- account-wide run records, if you can read the lakeflow system schema
SHOW TABLES IN system.lakeflow;

• Run history is kept for up to 60 days: export what you need to keep
• Add your own logging with print/logging, and write audit rows (run id, counts) to a Delta table

• Open the failed run → Repair run
• Only failed tasks and the ones downstream of them run again. Successful tasks are kept
• You can change parameters in the repair dialog. Settings changes you made first also apply
• Works for jobs with two or more tasks. For a single task, just Run now
• The run keeps one record, with a column for each repair in the matrix view

• Practice: make transform fail (rename the staging table), run, fix it, Repair run
• Repair reruns each task from its start: tasks must be idempotent

Run backfill: next to Run now → Run backfill.

• Databricks creates one run per interval, labelled Backfill
• Max 100 runs per backfill: use a bigger interval or several backfills
• Always runs the whole job: you can't pick tasks
• Raise max concurrent runs to run slices in parallel. The default is 1
• The job must read the date parameter and process only that slice

# use the date in the task, never "today"
run_date = dbutils.widgets.get("run_date")[:10]     # keep YYYY-MM-DD

• Use MERGE or overwrite-by-range (Module 8.13) so that the same slice twice gives the same table

• Two ways to combine them: Airflow triggers an existing Databricks job, or Airflow submits one-time runs
• If everything lives in Databricks, Lakeflow Jobs alone is simpler
• Use Airflow when a company standard or cross-system dependencies need it
• Jobs appear in the Databricks Runs tab even when Airflow started them
• Keep the schedule and the retries in one layer
• The hands-on version is Module 28