Skip to content
Tutorial Reference

How to Duplicate a Row N Times in a PySpark DataFrame

Learn efficient PySpark methods to duplicate rows dynamically based on column values or a fixed count, including explode(array_repeat()), union(), and RDD-based approaches.

When working with PySpark DataFrames, you may need to duplicate rows, whether for data augmentation, testing with larger datasets, generating repeated records based on a column value, or creating weighted samples. PySpark provides several approaches to replicate rows efficiently across distributed data.

In this guide, you'll learn multiple methods to duplicate rows in a PySpark DataFrame, from column-value-based repetition to fixed N-time duplication.

Setting Up the Example

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DuplicateRows").getOrCreate()

data = [
    ("Alice", 2, "Engineering"),
    ("Bob", 3, "Marketing"),
    ("Charlie", 1, "Sales"),
    ("Diana", 4, "Engineering")
]

columns = ["Name", "Repeat_Count", "Department"]
df = spark.createDataFrame(data, columns)

df.show()

Output:

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
+-------+------------+-----------+

Method 1: Repeating Rows Based on a Column Value

When you want each row duplicated a number of times specified by a column value, use array_repeat() with explode():

from pyspark.sql.functions import expr

df_repeated = df.withColumn(
    "Repeat_Count",
    expr("explode(array_repeat(Repeat_Count, int(Repeat_Count)))")
)

df_repeated.show()

Output:

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
+-------+------------+-----------+

How it works:

  1. array_repeat(Repeat_Count, int(Repeat_Count)) creates an array containing the value repeated N times (e.g., [2, 2] for Alice).
  2. explode() converts each array element into a separate row.
  3. The result is each row appearing as many times as specified by its Repeat_Count value.

Method 2: Duplicating All Rows a Fixed N Times

To duplicate every row the same number of times, create an array of size N and explode it:

from pyspark.sql.functions import explode, array_repeat, lit, col

n = 3  # Number of times to duplicate each row

df_duplicated = df.withColumn("temp", explode(array_repeat(lit(1), n)))
df_duplicated = df_duplicated.drop("temp")

df_duplicated.show()

Output:

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
+-------+------------+-----------+

Each row appears exactly 3 times. The temporary column created by explode() is dropped since it's not needed.

Method 3: Using union() in a Loop

A straightforward approach is to union the DataFrame with itself N times:

n = 3  # Number of total copies (original + duplicates)

df_result = df
for _ in range(n - 1):
    df_result = df_result.union(df)

print(f"Original rows: {df.count()}")
print(f"After duplication: {df_result.count()}")

df_result.orderBy("Name").show()

Output:

Original rows: 4
After duplication: 12

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
+-------+------------+-----------+

Method 4: Using flatMap() on RDD

For maximum flexibility, convert to RDD and use flatMap() to replicate rows:

n = 3

rdd_repeated = df.rdd.flatMap(lambda row: [row] * n)
df_result = spark.createDataFrame(rdd_repeated, df.schema)

df_result.show()

Output:

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
|  Diana|           4|Engineering|
+-------+------------+-----------+

This approach is simple and works well, but converting between DataFrame and RDD adds some overhead.

Method 5: Duplicating a Specific Row

To duplicate a specific row (not all rows), filter it first and then union:

from functools import reduce
from pyspark.sql import DataFrame

target_row = df.filter(df.Name == "Alice")
n = 4  # Number of copies

copies = [target_row] * n
duplicated_row = reduce(DataFrame.union, copies)

df_result = df.union(duplicated_row)

df_result.show()

Output:

+-------+------------+-----------+
|   Name|Repeat_Count| Department|
+-------+------------+-----------+
|  Alice|           2|Engineering|
|    Bob|           3|  Marketing|
|Charlie|           1|      Sales|
|  Diana|           4|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
|  Alice|           2|Engineering|
+-------+------------+-----------+

Alice's row now appears 5 times total (1 original + 4 copies).

Adding a Copy Identifier

To track which copy each row belongs to, add an index column during duplication:

from pyspark.sql.functions import explode, sequence, lit

n = 3

df_with_copy_id = df.withColumn(
    "copy_id",
    explode(sequence(lit(1), lit(n)))
)

df_with_copy_id.orderBy("Name", "copy_id").show()

Output:

+-------+------------+-----------+-------+
|   Name|Repeat_Count| Department|copy_id|
+-------+------------+-----------+-------+
|  Alice|           2|Engineering|      1|
|  Alice|           2|Engineering|      2|
|  Alice|           2|Engineering|      3|
|    Bob|           3|  Marketing|      1|
|    Bob|           3|  Marketing|      2|
|    Bob|           3|  Marketing|      3|
|Charlie|           1|      Sales|      1|
|Charlie|           1|      Sales|      2|
|Charlie|           1|      Sales|      3|
|  Diana|           4|Engineering|      1|
|  Diana|           4|Engineering|      2|
|  Diana|           4|Engineering|      3|
+-------+------------+-----------+-------+

The copy_id column identifies each copy (1 = original, 2 = first copy, etc.).

Quick Comparison of Methods

Method Fixed N Column-Based N Specific Row Performance
explode(array_repeat()) ✅ ✅ ❌ ⚡ Best
union() loop ✅ ❌ ✅ 🔶 OK for small N
flatMap() on RDD ✅ ✅ ❌ 🔶 Moderate (RDD overhead)
Filter + union ❌ ❌ ✅ 🔶 Moderate

Conclusion

PySpark provides several ways to duplicate rows in a DataFrame:

  • Use explode(array_repeat()) for the most efficient and scalable approach. It works natively with Spark's catalyst optimizer and handles both fixed N and column-based repetition.
  • Use union() in a loop for simple cases with small N values or when duplicating specific rows.
  • Use flatMap() on RDD when you need maximum flexibility in the replication logic.
  • Add a copy_id column with sequence() when you need to track which copy each row represents.

For most production use cases, the explode(array_repeat()) approach is the recommended method due to its performance and simplicity.

Navigation