Skip to content
Tutorial Reference

Python Pandas: How to Add a Level to a MultiIndex (Rows or Columns)

Learn how to add a new level to a Pandas MultiIndex for rows or columns using pd.concat() and pd.MultiIndex.from_product() with practical examples.

Pandas MultiIndex (also known as a hierarchical index) allows you to have multiple levels of indexing on your DataFrame's rows or columns, enabling more complex data organization and analysis. Sometimes, you may need to add a new level to an existing index or column structure.

This guide explains several methods to add a new level to a Pandas MultiIndex for both rows (index) and columns, primarily using pd.concat() and pd.MultiIndex.from_product().

Understanding MultiIndex and Levels

A MultiIndex can be thought of as an index with multiple "layers" or "levels." Each level can have its own name.

  • Row Index: If a DataFrame has a MultiIndex on its rows, each row is identified by a tuple of values, one from each level.
  • Column Index: Similarly, columns can have multiple levels, meaning each column header is a tuple of values.

Adding a level means introducing a new layer of grouping or categorization to your existing index or columns.

Example DataFrame (Single Index):

import pandas as pd

data = {
    ('Group1', 'Value1'): [10, 20, 30],
    ('Group1', 'Value2'): [15, 25, 35],
    ('Group2', 'Value1'): [100, 200, 300]
}
df_simple_cols = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50],
    'C': [100, 200, 300, 400, 500]
})

print("Original DataFrame (Simple Index):")
print(df_simple_cols)

Output:

Original DataFrame (Simple Index):
   A   B    C
0  1  10  100
1  2  20  200
2  3  30  300
3  4  40  400
4  5  50  500

Adding a Level to the DataFrame's Row Index

This adds an outer level to the existing row index.

The pd.concat() function can be used to add a new outermost level to the index by wrapping the DataFrame in a list and providing the keys and names arguments.

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50],
    'C': [100, 200, 300, 400, 500]
})

new_outer_level_key = 'Experiment_X'
new_outer_level_name = 'Experiment_Set'

df_multi_index_rows = pd.concat(
    [df],                           # DataFrame(s) to concatenate, wrapped in a list
    keys=[new_outer_level_key],     # Key(s) for the new outermost level
    names=[new_outer_level_name]    # Name(s) for the new level(s)
)

print("DataFrame with New Outer Row Index Level:")
print(df_multi_index_rows)
print()

print("New index names:")
print(df_multi_index_rows.index.names)

Output:

DataFrame with New Outer Row Index Level:
                  A   B    C
Experiment_Set              
Experiment_X   0  1  10  100
               1  2  20  200
               2  3  30  300
               3  4  40  400
               4  5  50  500

New index names:
['Experiment_Set', None]
  • [df]: You pass a list containing the DataFrame(s). If just one, it's [df].
  • keys=['X']: This provides the label(s) for the new outermost level. For each DataFrame in the input list, concat uses a corresponding key.
  • names=['First Level']: This assigns a name to the newly created level.
  • You can also pass a dictionary to pd.concat() where keys become the new outer index level: 
    df_multi_index_dict = pd.concat({new_outer_level_key: df}, names=[new_outer_level_name])
    This produces the same result as above.

Using pd.MultiIndex.from_frame() (More Manual Control)

This method gives you more control if you need to insert the new level at a specific position (not just outermost) or if the new level's values are more complex.

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50],
    'C': [100, 200, 300, 400, 500]
})

index_df = df.index.to_frame(name='OriginalIndex') # Give the original index level a name

new_level_values = ['SetA'] * 2 + ['SetB'] * 3 # Example: different values for the new level
new_level_name = 'New_Level'

index_df.insert(0, new_level_name, new_level_values)
print("DataFrame representing the new MultiIndex structure:")
print(index_df)
print()

df.index = pd.MultiIndex.from_frame(index_df)

print("DataFrame with New Index Level via from_frame():")
print(df)

Output:

DataFrame representing the new MultiIndex structure:
  New_Level  OriginalIndex
0      SetA              0
1      SetA              1
2      SetB              2
3      SetB              3
4      SetB              4

DataFrame with New Index Level via from_frame():
                         A   B    C
New_Level OriginalIndex            
SetA      0              1  10  100
          1              2  20  200
SetB      2              3  30  300
          3              4  40  400
          4              5  50  500

Adding a Level to the DataFrame's Column Index

This adds an outer level to the existing column headers.

Using pd.concat() with axis=1

Similar to adding a row index level, but you specify axis=1 to operate on columns.

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50],
    'C': [100, 200, 300, 400, 500]
})

new_outer_col_level_key = 'Metrics'
new_outer_col_level_name = 'Category'

df_multi_col = pd.concat(
    [df],
    keys=[new_outer_col_level_key],
    names=[new_outer_col_level_name],
    axis=1 # Operate on columns
)

print("DataFrame with New Outer Column Level:")
print(df_multi_col)
print()

print("New column MultiIndex:")
print(df_multi_col.columns)

Output:

DataFrame with New Outer Column Level:
Category Metrics         
               A   B    C
0              1  10  100
1              2  20  200
2              3  30  300
3              4  40  400
4              5  50  500

New column MultiIndex:
MultiIndex([('Metrics', 'A'),
            ('Metrics', 'B'),
            ('Metrics', 'C')],
           names=['Category', None])

Using pd.MultiIndex.from_product()

This is very useful for creating a new column MultiIndex when the new level is the same for all existing columns, or when you want to create a Cartesian product of levels.

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [10, 20, 30, 40, 50],
    'C': [100, 200, 300, 400, 500]
})

new_outer_level = ['Measurements']

df.columns = pd.MultiIndex.from_product(
    [new_outer_level, df.columns],          # Iterables for the levels
    names=['Data_Type', 'Original_Metric']  # Optional names for the levels
)

print("DataFrame with New Column Level via from_product():")
print(df)
print()

print("New column MultiIndex:")
print(df.columns)

Output:

DataFrame with New Column Level via from_product():
Data_Type       Measurements         
Original_Metric            A   B    C
0                          1  10  100
1                          2  20  200
2                          3  30  300
3                          4  40  400
4                          5  50  500

New column MultiIndex:
MultiIndex([('Measurements', 'A'),
            ('Measurements', 'B'),
            ('Measurements', 'C')],
           names=['Data_Type', 'Original_Metric'])

Conclusion

Adding levels to a Pandas MultiIndex allows for more sophisticated data structuring:

  • For Row Indexes:
    • pd.concat([df], keys=['NewOuterKey'], names=['NewLevelName']) is straightforward for adding a new outermost level.
    • Converting the index to a DataFrame (df.index.to_frame()), inserting the new level's data, and then converting back (pd.MultiIndex.from_frame()) offers more control over placement and values.
  • For Column Indexes:
    • pd.concat([df], keys=['NewOuterKey'], names=['NewLevelName'], axis=1) adds a new outermost column level.
    • pd.MultiIndex.from_product([['OuterLevelName'], df.columns]) is excellent for adding a uniform outer level to existing columns or creating new hierarchical structures.

Choose the method that best fits the complexity of the new level you're adding and whether you're targeting rows or columns.

Navigation