DatabricksZero to Advanced
/
Course/Level 1 · Foundations

Module 3: Apache Spark Fundamentals

The Spark ideas you need to read plans, reason about performance and avoid surprises on serverless.

6 hands-on1 with caveats5 concept12 topics
Topics in this module
  1. Spark Architecture
  2. Driver, Executors, and Workers
  3. SparkSession and SparkContext
  4. DataFrames vs RDDs
  5. Lazy Evaluation
  6. Transformations vs Actions
  7. Narrow vs Wide Transformations
  8. Jobs, Stages, and Tasks
  9. DAG and Execution Plans
  10. Partitions and Parallelism
  11. Spark SQL vs PySpark
  12. Spark Execution Lifecycle

The mental model behind Spark: where code runs, when work happens, what a shuffle costs.

  • Goal: read a query plan and predict performance
  • Examples use DataFrame and SQL only, so they work on Free Edition

Serverless

No RDD APIs, no sparkContext, caching APIs raise errors. Driver and executors are managed for you.

3.01Concept

Spark Architecture

By the end: draw the Spark architecture and say which part plans work and which part does it.

Your code notebook or job Driver plans and schedules Holds the SparkSession Builds the query plan Splits work into tasks Collects the results tasks results Worker nodes (managed by Databricks) Executor 1 task task holds partitions and shuffle files Executor 2 task task holds partitions and shuffle files Executor 3 task task holds partitions and shuffle files Cloud object storage: Delta and Parquet files read and write partitions
Driver plans and schedules. Executors run one task per partition and read or write files in cloud storage.
  • Driver: one machine. Holds the spark session, builds the plan, splits it into tasks, collects results
  • Executors: run tasks, one task per partition
  • Data stays in cloud storage: executors read it only while a task runs

Remember

Work is split by data (partitions). Anything pulled back to the driver must fit on one machine.

3.02Concept

Driver, Executors, and Workers

By the end: tell the driver, an executor, a worker and a task apart, and know which one runs your Python code.

Workera machine (VM)hostsExecutora process on the workerrunsTaskone operation on one partition
Nesting: workers host executors, executors run tasks.
  • Driver: runs your Python notebook code, builds plans. collect() here can run out of memory
  • Executor: holds shuffle files and cached data. Memory limits cause spills
  • Task: one slow task = skewed partition
  • DataFrame calls are descriptions of work. The heavy lifting runs in the engine, not in Python
  • Python UDFs run per row in Python workers, so they are slower

Best practice

Filter, join and aggregate with DataFrame functions. Bring only small results to the driver.

3.03Hands-on with caveats

SparkSession and SparkContext

By the end: use the `spark` session correctly, and replace SparkContext and RDD code with DataFrame code on serverless compute.

  • Every notebook already has spark. Do not build your own session
  • SparkContext (sc) and RDDs are not available on serverless
Python
print(spark.version)
print(spark.conf.get("spark.sql.session.timeZone"))
Old RDD code Use instead
sc.parallelize(data) spark.createDataFrame(data, schema)
sc.textFile(path) spark.read.text(path)
rdd.map(f) column expressions or pandas_udf
rdd.reduceByKey(f) df.groupBy(key).agg(...)
rdd.getNumPartitions() F.spark_partition_id() (topic 10)

Pitfall

SparkSession.builder.getOrCreate() and .config(...) from tutorials: skip them. Most configs can't be set on serverless.

3.04Concept

DataFrames vs RDDs

By the end: explain why DataFrames replaced RDDs for almost all work.

RDDopaque objectsno optimiserPython functionsnot on serverlessDataFramenamed, typed columnsCatalyst optimiserbuilt-in engine opsworks everywhere
  • Spark can't see inside an RDD function, so it can't optimise it
  • Because a DataFrame has a schema, Spark can reorder and prune the plan
Python
from pyspark.sql import functions as F

lines = spark.createDataFrame([("to be or not to be",), ("to do is to be",)], ["line"])

(lines.select(F.explode(F.split("line", " ")).alias("word"))
      .groupBy("word").count()
      .orderBy(F.desc("count")).limit(2).show())
Output
+----+-----+
|word|count|
+----+-----+
|  to|    4|
|  be|    3|
+----+-----+
3.05Hands-on on Free Edition

Lazy Evaluation

By the end: see lazy evaluation for yourself: transformations return immediately, an action does the work.

filterlazywithColumnlazycount()actiontriggersWork runsread, filter, count
Nothing runs until an action is called.
Python
import time
from pyspark.sql import functions as F

t = time.time()
df = (spark.range(10_000_000)
        .filter("id % 2 = 0")
        .withColumn("square", F.col("id") * F.col("id")))
print(f"plan: {time.time() - t:.2f}s")      # instant

t = time.time()
print(f"count: {df.count():,} in {time.time() - t:.2f}s")   # 5,000,000
  • Laziness lets Spark see the whole chain: prune columns, push filters down, combine steps
  • Every action recomputes the chain. Serverless can't cache: write expensive results to a Delta table

Errors can appear late

Spark Connect resolves names at execution time. A bad column name may fail at the action, not at the line that caused it.

3.06Hands-on on Free Edition

Transformations vs Actions

By the end: tell transformations from actions and pick safe actions for exploring.

Transformations (lazy)select, filterwithColumn, dropgroupBy().agg()join, unionorderBy, distinctActions (run work)show(), display()count()first(), take(n)collect(), toPandas()write, saveAsTable()
Python
df = spark.range(5)

df.show()
print(df.count())      # 5
print(df.first())      # Row(id=0)
print(df.take(3))      # 3 Rows

Pitfall

collect() and toPandas() copy everything to the driver. Use limit or aggregate first.

3.07Hands-on on Free Edition

Narrow vs Wide Transformations

By the end: tell narrow from wide transformations and spot the shuffle in a plan.

Stage 1 narrow: filter, select task 1: partition 1 task 2: partition 2 task 3: partition 3 stage boundary = shuffle rows move by key Stage 2 wide: groupBy, count task 1: keys A to M task 2: keys N to Z
A shuffle sends rows between executors by key. It is the boundary between stage 1 and stage 2.
  • Narrow: each output partition needs one input partition. filter, select, withColumn
  • Wide: needs rows from many partitions, so Spark shuffles. groupBy, join, distinct
  • A shuffle writes to disk, crosses the network, reads back: the expensive part
Python
from pyspark.sql import functions as F

df = spark.range(1_000_000).withColumn("bucket", F.col("id") % 10)

df.filter(F.col("id") > 10).select("id", "bucket").explain()   # no Exchange
df.groupBy("bucket").count().explain()                          # has Exchange
  • In the plan, look for Exchange hashpartitioning(...): that is the shuffle

Design rule

Narrow work first: filter early, select only needed columns. Shuffle as little as you can.

3.08Concept

Jobs, Stages, and Tasks

By the end: explain how one action becomes jobs, stages and tasks.

Actione.g. count(), writecreatesJoball work for that actionsplit at each shuffleStagesa group of tasks with no shuffle insideone per partitionTasksrun on executors
  • Example: groupBy = 2 stages. With 3 input and 2 shuffle partitions: 3 tasks, then 2 tasks
  • Where to inspect jobs, stages and tasks depends on compute type (Spark UI, query profile: Module 12)

Two kinds of job

A Spark job is the work of one action. A Lakeflow Job (Module 10) is a scheduled workflow. Unrelated.

3.09Hands-on on Free Edition

DAG and Execution Plans

By the end: read a query plan with `explain` and say what each stage of planning does.

Parsedas writtenAnalyzednames resolvedOptimizedpushdown, pruningPhysicalhow it runs
explain(True) prints all four plans.
Python
from pyspark.sql import functions as F

q = (spark.read.table("samples.nyctaxi.trips")
       .filter("trip_distance > 5")
       .groupBy("pickup_zip")
       .agg(F.avg("fare_amount").alias("avg_fare")))

q.explain("formatted")
  • No samples catalog? Use any table of yours (Module 4 creates some)
  • Read the plan from source up to result. Look for:
    • Scan with filters pushed down
    • Filter and Project (columns kept)
    • Exchange = shuffle
    • Aggregate twice: partial before the shuffle, final after
3.10Hands-on on Free Edition

Partitions and Parallelism

By the end: see how rows spread across partitions and when to use `repartition` and `coalesce`.

Python
from pyspark.sql import functions as F

df = spark.range(1_000_000)

(df.withColumn("pid", F.spark_partition_id())
   .groupBy("pid").count().orderBy("pid").show())   # rows per partition

more = df.repartition(8)          # full shuffle
fewer = more.coalesce(2)          # merge, no full shuffle
repartition(n)full shufflemore or rebalancedrepartition(col)full shufflegroup rows by keycoalesce(n)no full shufflefewer partitions only
  • Counts per partition depend on your compute, so your output will differ
  • getNumPartitions() is RDD-only, so use spark_partition_id() on serverless

More partitions is not faster

Tiny partitions = overhead and small files. Huge ones = slow, memory-hungry tasks. Repartition only with a measured reason.

3.11Hands-on on Free Edition

Spark SQL vs PySpark

By the end: write one query in SQL and in PySpark and explain why they perform the same.

SQL or DataFrameyour codeLogical plansame for bothCatalystoptimiserPhysical plansame engine
Both front ends end up as the same plan. Choose by readability.
SQL
%sql
SELECT pickup_zip, AVG(fare_amount) AS avg_fare
FROM samples.nyctaxi.trips
WHERE trip_distance > 5
GROUP BY pickup_zip
Python
from pyspark.sql import functions as F

display(spark.read.table("samples.nyctaxi.trips")
        .filter(F.col("trip_distance") > 5)
        .groupBy("pickup_zip")
        .agg(F.avg("fare_amount").alias("avg_fare")))
  • Run explain() on both: same operators
  • SQL: set-based, readable for analysts, good for reports and declarative pipelines
  • PySpark: reusable functions, loops, unit tests
  • Mix freely: df.createOrReplaceTempView(...) and spark.sql(...) (Module 4)
3.12Concept

Spark Execution Lifecycle

By the end: describe in order what happens between pressing Shift+Enter and getting a result.

an action triggers it Your codeDataFrame or SQL Logical planwhat you asked Analyzed plannames resolved Optimized planCatalyst rewrites Physical planhow it runs Stages, tasksrun on executors planning: no rows are processed yet execution: data moves
Planning processes no rows. Execution is where data is read, tasks run and shuffles happen.
  1. Action called: nothing ran before this
  2. Logical plan built from your code
  3. Analysed: names and types resolved
  4. Optimised by Catalyst: pushdown, pruning
  5. Physical plan chosen (e.g. join strategy)
  6. Cut into stages at shuffles, then into tasks
  7. Executors run tasks, results return or are written

Remember

Data is read only in step 7. Steps 2 to 6 are cheap planning.

Recap

What you can do now

  • Driver plans, executors run tasks, one task per partition
  • Transformations are lazy, actions run work
  • Shuffles (Exchange) split jobs into stages
  • Read explain output; inspect partitions with spark_partition_id()
  • On serverless: no SparkContext, RDDs or caching