Module 27: Run the Project with Databricks Jobs

Steps:

1. Sidebar Jobs & Pipelines → Create → Job. Name it orders-pipeline
2. Task name load_silver. Type Python wheel
3. Package name orders_pipeline. Entry point load_silver
4. Compute: Serverless. Under Environment and libraries, add the wheel path /Volumes/<catalog>/dbx_project/artifacts/orders_pipeline-0.1.0-py3-none-any.whl
5. Parameters, keyword: catalog = your catalog, schema = dbx_project, run_date = 2025-03-15
6. Save, then Run now

# check the result in a notebook
spark.sql("SELECT count(*) FROM dbx_project.silver_orders").show()

• Wheel tasks send parameters as command-line arguments. If the run fails with an argument error, open the run output and compare it with parse_args (Module 26.5)
• Dependencies for a wheel on serverless go in the environment, not a cluster library list
• Use a service principal as run-as for production (Module 17). For learning, your own identity is fine

• Add task each time, same wheel, a different entry point. Set Depends on in the task panel
• A failing check_quality raises (Module 26.9), so build_gold is skipped. Gold never sees bad silver
• By default a task needs all upstream tasks to succeed. Run if changes that: at least one succeeded, none failed, at least one failed, all done
• A cleanup or notify task with "at least one failed" is the pattern for a failure hook

Try it: run with a bad quantity row injected into silver_orders after load_silver. See build_gold skipped.

• Tasks with no dependencies run in parallel (limited to 5 at once on Free Edition)

Edit each task's parameters to the references:

catalog   = {{job.parameters.catalog}}
schema    = {{job.parameters.schema}}
run_date  = {{job.parameters.run_date}}
days      = {{job.parameters.days}}

• Set the job parameter run_date empty for scheduled runs: the code uses today (UTC). Backfills and reruns pass an explicit date
• Parameters are strings: --days is converted by argparse (type=int)
• If a job parameter and a task parameter have the same key, the job parameter wins
• Run now with different settings overrides parameters for one run

Schedule: job page → Schedules & Triggers → Add trigger → Scheduled. Pick a time, a timezone, save.

• Use an explicit timezone. UTC avoids daylight-saving surprises
• Pause the schedule instead of deleting it while debugging
• Set max concurrent runs = 1 so a slow run is not overlapped by the next
• Free Edition: scheduled jobs work. Continuous and large file-arrival setups: check what your workspace allows

• Retry only for transient causes (network, a busy service). A retry that repeats a code error wastes compute
• Retries are safe only if the task is idempotent: this project overwrites or replaceWheres, so yes
• Add yourself to failure notifications. A job that fails silently is worse than no job
• Email is the simplest on Free Edition. Slack or webhook destinations need to be set up first

Where to look:

%sql
-- optional: every job run in one table (system tables; availability varies)
SELECT job_id, period_start_time, result_state
FROM system.lakeflow.job_run_timeline
ORDER BY period_start_time DESC
LIMIT 20

• Run history is kept about 60 days: copy what you need longer into your own ops_run_log (Module 16)
• Failure in logs of one task, not of the job. Start from the red box
• Compare a slow run with a fast one: same parameters? same data volume?

Repair run:

• Run page → Repair run. It reruns failed and skipped tasks plus their dependents. Successful tasks are not rerun
• It applies the current job settings: fix the cause first (parameter, wheel version, data)
• Repair is for jobs with 2 or more tasks. For a one-task job use Run now
• Databricks does not make tasks idempotent. Your tasks must be: ours overwrite, so a repair can't duplicate

Backfill: run the job for past dates.

Try it: run for 2025-03-10 through 2025-03-12, then check:

%sql
SELECT order_date, orders, revenue
FROM dbx_project.gold_daily_revenue
WHERE order_date BETWEEN '2025-03-10' AND '2025-03-12'
ORDER BY order_date

• days defaults to 7 so each run rewrites a window. Late data inside that window is picked up automatically

# databricks.yml
bundle:
  name: orders-pipeline

variables:
  catalog:
    description: Unity Catalog catalog
  schema:
    default: dbx_project

artifacts:
  orders_wheel:
    type: whl
    build: python -m build --wheel
    path: .

resources:
  jobs:
    orders_pipeline:
      name: orders-pipeline-${bundle.target}
      max_concurrent_runs: 1
      parameters:
        - { name: catalog,  default: "${var.catalog}" }
        - { name: schema,   default: "${var.schema}" }
        - { name: run_date, default: "" }
        - { name: days,     default: "7" }
      schedule:
        quartz_cron_expression: "0 30 6 * * ?"
        timezone_id: UTC
        pause_status: PAUSED
      email_notifications:
        on_failure: ["you@example.com"]
      environments:
        - environment_key: default
          spec:
            environment_version: "3"              # copy the current value from `databricks bundle init`
            dependencies:
              - ./dist/*.whl
      tasks:
        - task_key: load_silver
          environment_key: default
          max_retries: 2
          python_wheel_task:
            package_name: orders_pipeline
            entry_point: load_silver
            named_parameters: { catalog: "{{job.parameters.catalog}}", schema: "{{job.parameters.schema}}", run_date: "{{job.parameters.run_date}}", days: "{{job.parameters.days}}" }
        - task_key: check_quality
          depends_on: [{ task_key: load_silver }]
          environment_key: default
          python_wheel_task:
            package_name: orders_pipeline
            entry_point: check_quality
            named_parameters: { catalog: "{{job.parameters.catalog}}", schema: "{{job.parameters.schema}}", run_date: "{{job.parameters.run_date}}", days: "{{job.parameters.days}}" }
        - task_key: build_gold
          depends_on: [{ task_key: check_quality }]
          environment_key: default
          python_wheel_task:
            package_name: orders_pipeline
            entry_point: build_gold
            named_parameters: { catalog: "{{job.parameters.catalog}}", schema: "{{job.parameters.schema}}", run_date: "{{job.parameters.run_date}}", days: "{{job.parameters.days}}" }

targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: workspace            # use the result of SELECT current_catalog()
  prod:
    mode: production
    variables:
      catalog: prod_catalog
    workspace:
      host: https://<prod-workspace-url>

databricks bundle validate -t dev
databricks bundle deploy -t dev
databricks bundle run orders_pipeline -t dev

• Development mode prefixes the job name and pauses its schedule: unpause only in prod
• The wheel is built by the artifacts step and uploaded on deploy: you don't upload it by hand
• Check environment_version and the wheel dependencies path against a template from databricks bundle init: they change between versions
• Production mode also wants run_as and permissions: add a service principal (Module 15)
• Edit in the UI only to experiment. The repo is the source of truth

# .github/workflows/deploy.yml
name: deploy
on:
  push:
    branches: [main]
concurrency: deploy-prod
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install build pytest && pip install -e .
      - run: pytest tests/unit
      - uses: databricks/setup-cli@main
      - run: databricks bundle validate -t prod
        env:
          DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}
      - run: databricks bundle deploy -t prod
        env:
          DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }}

• CI uses a service principal token, stored in the CI secrets. Not your personal token
• Run unit tests without Databricks first (local Spark): they need no secrets
• Bump version in pyproject.toml per release so the new wheel has a new name
• Rollback: deploy the previous git tag
• A smoke run after deploy (databricks bundle run ... -t dev) catches a broken wheel before prod
• Module 28 triggers this same job from Airflow without changing it