Module 22: Industry Standards & Best Practices

• From the docs, operational excellence means: version control, CI/CD, environment isolation by workspace or catalog, managed tables, IaC, monitoring
• Reliability means: Delta for ACID, retry policies and maximum runtimes on jobs, quality checks at ingest, tested recovery (checkpoints, time travel, repair runs)
• Use the pillars as headings in design reviews and post-incident reviews (Module 19.12)

Before:

d = spark.table("orders")
d = d.filter("status = 'completed' or status = 'shipped'")
d = d.withColumn("r", d.quantity * 2)
d = d.groupBy("customer_id").agg(F.sum("r").alias("x"))

After:

REVENUE_STATUSES = ["completed", "shipped"]

def completed_quantity_by_customer(orders: DataFrame) -> DataFrame:
    """Total ordered quantity per customer, counting only revenue-bearing orders."""
    return (
        orders
        .filter(F.col("status").isin(REVENUE_STATUSES))
        .groupBy("customer_id")
        .agg(F.sum("quantity").alias("total_quantity"))
    )

result = completed_quantity_by_customer(spark.table("orders"))

• Format with a tool (black/ruff format) and lint (ruff): style is not a debate
• Small functions: if you can't name it in a few words, split it

# dependency inversion: the function gets what it needs
def build_daily(orders: DataFrame, products: DataFrame) -> DataFrame:
    return (orders.join(products, "product_id")
                  .groupBy(F.to_date("order_ts").alias("order_date"))
                  .agg(F.round(F.sum(F.col("quantity") * F.col("unit_price")), 2).alias("revenue")))

# the caller decides where the data comes from and goes to
daily = build_daily(spark.table("orders"), spark.table("products"))

• The practical test: can I unit-test this function with two tiny DataFrames? If yes, the design is probably sound
• Don't over-engineer: patterns are for repeated problems

def read_orders(spark, table: str) -> DataFrame:
    return spark.table(table)

def transform(orders: DataFrame) -> DataFrame:
    return completed_quantity_by_customer(orders)

def validate(df: DataFrame) -> DataFrame:
    assert df.filter("total_quantity <= 0").count() == 0, "non-positive totals"
    return df

def write(df: DataFrame, table: str) -> None:
    df.write.mode("overwrite").saveAsTable(table)

def run(spark, source: str, target: str) -> None:
    write(validate(transform(read_orders(spark, source))), target)

run(spark, "orders", "std_customer_quantity")

• A new source or target changes one function
• Packages, wheels and project layout: Module 26

• Business logic doesn't know about tables, paths or schedules
• I/O code doesn't contain business rules
• Environment values (catalog, paths, secrets) are never inside the logic
• Scheduling lives in the job definition, not in sleep loops in notebooks
• Platform setup (compute, grants) is in infrastructure code, not in the pipeline
• Separate concerns = each can change and be tested alone

%sql
ALTER TABLE orders_delta SET TBLPROPERTIES ('quality.owner' = 'sales-data', 'quality.sla_hours' = '24')

• A data quality score ("98.7% of rows passed") is a useful trend metric, never the whole story
• Quality rules are code: reviewed, tested, versioned (Module 14)
• Fix problems at the source where you can

• Lowercase and underscores: avoids quoting and clashes
• Use UTC for timestamps, and say so
• Consistent beats clever. Write the rules in one page and put it in the repo
• Rename with a view or alias and a deprecation period

• Never edit the underlying files of a Delta table
• Use CREATE OR REPLACE TABLE to rebuild, not delete and recreate
• Don't use Spark caching on tables: it loses skipping and can show stale data

• Databricks recommends liquid clustering for new tables, together with managed tables and predictive optimization
• Clustering keys: the columns you filter on most (up to 4). Not compatible with partitioning or ZORDER
• Partition only very large tables and low-cardinality columns (such as date) where each partition is still large
• Over-partitioning (thousands of tiny partitions) is worse than none
• Run OPTIMIZE regularly or let predictive optimization do it. OPTIMIZE compacts files. VACUUM removes old ones
• Don't call REFRESH TABLE or manage partitions by hand: filter with WHERE

• Alert on symptoms people feel (stale dashboard) and causes you can act on (failed task)
• Every alert has an action. If it doesn't, delete it
• Retention: keep audit and run logs as long as compliance needs
• Detail: Module 16

• Unity Catalog for all data. Three-level names always
• Groups, not individuals, in grants. Least privilege. Owners are groups
• Service principals run production jobs
• Secrets in scopes (or a secret manager). Never in code, notebooks, logs, parameters
• Separate environments by catalog, schema or workspace
• Sensitive data tagged, masked or filtered at the table (row filters, column masks, ABAC)
• Retention and deletion policies written down and implemented (DELETE, REORG, VACUUM)
• Audit access and changes. Review grants regularly
• Network controls where regulation needs them (private connectivity, egress control)
• Governance changes go through code review

%sql
-- a standing review query: who holds powerful privileges on this schema?
SHOW GRANTS ON SCHEMA dbx_course

• Pre-commit hooks for format, lint and secret scanning
• Keep notebooks small and clean in Git: no outputs committed
• Detail: Modules 14, 15, 26

Checklist. A pipeline is ready for production when each line is true:

Code
 [ ] In Git, reviewed, tagged with a version
 [ ] Unit tests pass. Integration test run on test data
 [ ] No hard-coded paths, catalogs, secrets or user names
Data
 [ ] Output grain, schema and meaning documented. Table owner set
 [ ] Quality rules defined with actions. Quarantine in place
 [ ] Idempotent: a rerun gives the same result. Backfill tested
Operations
 [ ] Runs as a job with a service principal. Schedule and parameters set
 [ ] Retries, timeouts and failure notifications set
 [ ] Alerts for failure, duration and freshness
 [ ] Runbook written. On-call named
Governance and cost
 [ ] Permissions granted to groups, least privilege
 [ ] Sensitive columns tagged and protected
 [ ] Tags for cost attribution. Cost estimate known
Recovery
 [ ] Time travel / restore plan tested
 [ ] Rollback = previous tag redeployed. Tried once

• Run the list in a review meeting, not alone
• Keep it short enough to be used. Remove lines nobody checks