PySpark DataFrames support nested columns - columns that contain structured sub-fields defined using StructType. These are common when working with JSON data, complex schemas, or hierarchical information like addresses, names, or dates. While creating nested columns is straightforward, updating individual sub-fields within a nested column requires specific techniques.
In this guide, you will learn what nested columns are, how to create them, and how to update their values - both unconditionally and based on conditions - using PySpark's withField(), col(), lit(), and when() functions.
What Are Nested Columns?
A nested column (also called a struct column) is a column that contains multiple sub-fields. In PySpark, these are defined using StructType with individual sub-fields specified as StructField entries.
Real-world examples of nested columns:
| Nested Column | Sub-fields |
|---|---|
full_name |
first_name, middle_name, last_name |
address |
street, city, state, zip_code |
date_of_birth |
year, month, day |
You access sub-fields using dot notation: col("full_name.first_name").
Setting Up the Example DataFrame
Let's create a DataFrame with a nested Date_Of_Birth column containing Year, Month, and Date sub-fields:
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType
spark = SparkSession.builder.getOrCreate()
data = [
((2000, 21, 2), 18),
((1998, 14, 6), 24),
((1998, 1, 11), 18),
((2006, 30, 3), 16),
]
schema = StructType([
StructField("Date_Of_Birth", StructType([
StructField("Year", IntegerType(), True),
StructField("Month", IntegerType(), True),
StructField("Date", IntegerType(), True),
])),
StructField("Age", IntegerType(), True),
])
df = spark.createDataFrame(data=data, schema=schema)
df.show()
df.printSchema()Output:
+--------------+---+
| Date_Of_Birth|Age|
+--------------+---+
| {2000, 21, 2}| 18|
| {1998, 14, 6}| 24|
| {1998, 1, 11}| 18|
| {2006, 30, 3}| 16|
+--------------+---+
root
|-- Date_Of_Birth: struct (nullable = true)
| |-- Year: integer (nullable = true)
| |-- Month: integer (nullable = true)
| |-- Date: integer (nullable = true)
|-- Age: integer (nullable = true)Updating a Nested Column Unconditionally
To replace the value of a nested sub-field for all rows, use withColumn() combined with col().withField() and lit():
from pyspark.sql.functions import col, lit
updated_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth").withField("Year", lit(2024))
)
updated_df.show()Output:
+--------------+---+
| Date_Of_Birth|Age|
+--------------+---+
| {2024, 21, 2}| 18|
| {2024, 14, 6}| 24|
| {2024, 1, 11}| 18|
| {2024, 30, 3}| 16|
+--------------+---+Updating a Nested Column Conditionally
In most real-world scenarios, you want to update a nested sub-field only when a specific condition is met. Use the when().otherwise() construct to apply conditional logic.
Example 1: Update Based on the Nested Column's Own Value
Update the Date sub-field: if its current value is 2, replace it with 24; otherwise, keep the original value.
from pyspark.sql.functions import col, lit, when
updated_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth").withField(
"Date",
when(col("Date_Of_Birth.Date") == 2, lit(24))
.otherwise(col("Date_Of_Birth.Date"))
)
)
updated_df.show()Output:
+--------------+---+
| Date_Of_Birth|Age|
+--------------+---+
|{2000, 21, 24}| 18|
| {1998, 14, 6}| 24|
| {1998, 1, 11}| 18|
| {2006, 30, 3}| 16|
+--------------+---+Only the first row's Date changed from 2 to 24. All other rows retained their original values.
Example 2: Update Based on Another Column's Value
Update the Year sub-field: if the Age column equals 18, set Year to 2004; otherwise, keep the original value.
from pyspark.sql.functions import col, lit, when
updated_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth").withField(
"Year",
when(col("Age") == 18, lit(2004))
.otherwise(col("Date_Of_Birth.Year"))
)
)
updated_df.show()Output:
+-------------+---+
|Date_Of_Birth|Age|
+-------------+---+
|{2004, 21, 2}| 18|
|{1998, 14, 6}| 24|
|{2004, 1, 11}| 18|
|{2006, 30, 3}| 16|
+-------------+---+Rows 1 and 3 (where Age == 18) had their Year updated to 2004.
Example 3: Update Multiple Nested Sub-Fields
You can chain multiple withField() calls to update several sub-fields at once:
from pyspark.sql.functions import col, lit, when
updated_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth")
.withField(
"Year",
when(col("Age") == 18, lit(2004))
.otherwise(col("Date_Of_Birth.Year"))
)
.withField(
"Month",
when(col("Date_Of_Birth.Month") > 12, lit(12))
.otherwise(col("Date_Of_Birth.Month"))
)
)
updated_df.show()Output:
+-------------+---+
|Date_Of_Birth|Age|
+-------------+---+
|{2004, 12, 2}| 18|
|{1998, 12, 6}| 24|
| {2004, 1, 11}| 18|
|{2006, 12, 3}| 16|
+-------------+---+Both Year and Month were updated in a single transformation.
Common Mistake: Forgetting the otherwise() Clause
A frequent error when using conditional updates is omitting the otherwise() clause. Without it, rows that don't match the condition will have the sub-field set to null.
Wrong - missing otherwise():
wrong_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth").withField(
"Year",
when(col("Age") == 18, lit(2004))
)
)
wrong_df.show()Output:
+--------------+---+
| Date_Of_Birth|Age|
+--------------+---+
| {2004, 21, 2}| 18|
| {null, 14, 6}| 24|
| {2004, 1, 11}| 18|
| {null, 30, 3}| 16|
+--------------+---+Correct - always include otherwise() to preserve existing values:
correct_df = df.withColumn(
"Date_Of_Birth",
col("Date_Of_Birth").withField(
"Year",
when(col("Age") == 18, lit(2004))
.otherwise(col("Date_Of_Birth.Year")) # Preserve original value
)
)Alternative for Spark Versions Before 3.1
If you are using a Spark version older than 3.1 where withField() is not available, you can reconstruct the entire struct using the struct() function:
from pyspark.sql.functions import col, lit, when, struct
updated_df = df.withColumn(
"Date_Of_Birth",
struct(
when(col("Age") == 18, lit(2004))
.otherwise(col("Date_Of_Birth.Year")).alias("Year"),
col("Date_Of_Birth.Month").alias("Month"),
col("Date_Of_Birth.Date").alias("Date"),
)
)
updated_df.show()
updated_df.printSchema()Output:
+-------------+---+
|Date_Of_Birth|Age|
+-------------+---+
|{2004, 21, 2}| 18|
|{1998, 14, 6}| 24|
|{2004, 1, 11}| 18|
|{2006, 30, 3}| 16|
+-------------+---+
root
|-- Date_Of_Birth: struct (nullable = false)
| |-- Year: integer (nullable = true)
| |-- Month: integer (nullable = true)
| |-- Date: integer (nullable = true)
|-- Age: integer (nullable = true)Reading Nested Column Values
To access or filter on nested column values, use dot notation with col():
df.select(
col("Date_Of_Birth.Year").alias("Birth_Year"),
col("Date_Of_Birth.Month").alias("Birth_Month"),
col("Age"),
).show()
df.filter(col("Date_Of_Birth.Year") == 1998).show()Output:
+----------+-----------+---+
|Birth_Year|Birth_Month|Age|
+----------+-----------+---+
| 2000| 21| 18|
| 1998| 14| 24|
| 1998| 1| 18|
| 2006| 30| 16|
+----------+-----------+---+
+--------------+---+
| Date_Of_Birth|Age|
+--------------+---+
| {1998, 14, 6}| 24|
| {1998, 1, 11}| 18|
+--------------+---+Summary
Updating nested columns in PySpark requires a combination of withColumn(), col().withField(), and conditional logic. Here are the key takeaways:
| Task | Method |
|---|---|
| Update a sub-field for all rows | col("struct").withField("sub_field", lit(value)) |
| Update conditionally | when(condition, lit(new_value)).otherwise(col("struct.sub_field")) |
| Update multiple sub-fields | Chain multiple .withField() calls |
Spark < 3.1 (no withField) |
Reconstruct with struct() listing all sub-fields |
| Access nested values | col("struct.sub_field") |
- Always include
.otherwise()to preserve original values for non-matching rows. - Use dot notation (
col("Date_Of_Birth.Year")) to access nested sub-fields in conditions and selections. - Prefer
withField()(Spark 3.1+) overstruct()reconstruction for cleaner, more maintainable code.