Skip to content
Tutorial Reference

Python Pandas: Filter DataFrame Rows Using Regular Expressions (Regex)

Learn how to filter Pandas DataFrame rows using regex with str.contains() for complex string pattern matching, including case-insensitive searches and negation.

Filtering rows in a Pandas DataFrame based on complex string patterns is a common data manipulation task. While simple substring checks can be done with methods like str.startswith() or str.endswith(), regular expressions (regex) offer much more power and flexibility for matching intricate patterns within your string data.

This guide explains how to use the Series.str.contains() method with regular expressions to filter DataFrame rows, including case-insensitive matching and negation.

Why Use Regex for Filtering Pandas Rows?

Regular expressions allow you to define sophisticated search patterns that go beyond simple literal string matching. You can:

  • Match specific character sets (e.g., digits, letters, whitespace).
  • Define positions (start/end of string).
  • Specify repetitions (e.g., one or more occurrences).
  • Use OR conditions within a pattern.
  • Capture groups.

This makes regex invaluable when your filtering criteria are complex.

Example DataFrame:

import pandas as pd
import re # For re.IGNORECASE flag

data = {
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
}
df = pd.DataFrame(data)
print("Original DataFrame:")
print(df)

Output:

Original DataFrame:
      ID  Product Name                      Description
0   X101  Alpha Widget        High-quality alpha series
1  Y203A   Beta Gadget  Advanced beta model with extras
2  X105B   alpha gizmo         Compact gizmo for Alphas
3   Z300    Gamma Unit         Standard gamma component
4  Y203C  Delta Device            Next-gen delta system

The Core Method: Series.str.contains(regex_pattern)

To filter rows using regex, you apply the .str.contains() method to the specific Pandas Series (column) you want to test.

  1. Access the column: df['YourColumnName'].
  2. Use the .str accessor to use string methods.
  3. Call .contains(pattern, case=True, flags=0, na=None, regex=True).
    • pattern: Your regular expression string.
    • case: Boolean, default True (case-sensitive). Set to False for case-insensitive matching.
    • flags: Integer, regex compilation flags (e.g., re.IGNORECASE).
    • na: Value to return for missing (NaN) values in the Series.
    • regex: Boolean, default True. If False, pattern is treated as a literal string.

This method returns a boolean Series (True where the pattern matches, False otherwise). You then use this boolean Series to index the original DataFrame.

Filtering Rows Where a Column Matches a Regex

Example: Rows Starting With a Pattern

Let's select rows where the 'Product Name' starts with "Alpha".

import pandas as pd

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})


regex_starts_with_Alpha = r'^Alpha'

mask_starts_with_Alpha = df['Product Name'].str.contains(regex_starts_with_Alpha)
print("Boolean mask for 'starts with Alpha':")
print(mask_starts_with_Alpha)

df_starts_with_Alpha = df[mask_starts_with_Alpha]

print("Rows where 'Product Name' starts with 'Alpha':")
print(df_starts_with_Alpha)

Output:

Boolean mask for 'starts with Alpha':
0     True
1    False
2    False
3    False
4    False
Name: Product Name, dtype: bool
Rows where 'Product Name' starts with 'Alpha':
     ID  Product Name                Description
0  X101  Alpha Widget  High-quality alpha series

Case-Insensitive Matching (case=False or flags=re.IGNORECASE)

To match "Alpha" or "alpha":

  • Using case=False:

    import pandas as pd

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

regex_starts_with_alpha_case_insensitive = r'^alpha' # Pattern is lowercase

df_starts_alpha_case_false = df[df['Product Name'].str.contains(
    regex_starts_with_alpha_case_insensitive, case=False
)]
print("Rows where 'Product Name' starts with 'alpha' (case=False):")
print(df_starts_alpha_case_false)

    Output:

    Rows where 'Product Name' starts with 'alpha' (case=False):
    ID  Product Name                Description
0   X101  Alpha Widget  High-quality alpha series
2  X105B   alpha gizmo   Compact gizmo for Alphas

  • Using flags=re.IGNORECASE: (Requires import re)

    import pandas as pd
import re # Import the re module for flags

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

regex_starts_with_alpha_any_case = r'^alpha' # Pattern can be any case here too

df_starts_alpha_re_flag = df[df['Product Name'].str.contains(
    regex_starts_with_alpha_any_case, flags=re.IGNORECASE
)]
print("Rows where 'Product Name' starts with 'alpha' (flags=re.IGNORECASE):")
print(df_starts_alpha_re_flag)

    Output: (Same as with case=False)

    Rows where 'Product Name' starts with 'alpha' (flags=re.IGNORECASE):
    ID  Product Name                Description
0   X101  Alpha Widget  High-quality alpha series
2  X105B   alpha gizmo   Compact gizmo for Alphas

Filtering Rows Where a Column Does NOT Match a Regex (Negation)

To select rows that do not match the regex pattern, apply the tilde operator (~) to the boolean mask generated by str.contains(). This inverts the boolean Series.

import pandas as pd

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

regex_starts_with_Alpha = r'^Alpha'
mask_starts_with_Alpha = df['Product Name'].str.contains(regex_starts_with_Alpha)

df_not_starts_with_Alpha = df[~mask_starts_with_Alpha]

print("Rows where 'Product Name' does NOT start with 'Alpha':")
print(df_not_starts_with_Alpha)

Output:

Rows where 'Product Name' does NOT start with 'Alpha':
      ID  Product Name                      Description
1  Y203A   Beta Gadget  Advanced beta model with extras
2  X105B   alpha gizmo         Compact gizmo for Alphas
3   Z300    Gamma Unit         Standard gamma component
4  Y203C  Delta Device            Next-gen delta system

Alternative: Using Built-in String Methods (When Regex is Overkill)

For simple checks like "starts with," "ends with," or "contains literal substring," using dedicated Pandas string methods can be more readable and potentially slightly more performant than a general regex.

import pandas as pd

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

df_startswith_builtin = df[df['Product Name'].str.startswith('Alpha')]
print("Using str.startswith('Alpha'):")
print(df_startswith_builtin)
print()

df_endswith_builtin = df[df['Description'].str.endswith('extras')]
print("Using str.endswith('extras'):")
print(df_endswith_builtin)

Output:

Using str.startswith('Alpha'):
     ID  Product Name                Description
0  X101  Alpha Widget  High-quality alpha series

Using str.endswith('extras'):
      ID Product Name                      Description
1  Y203A  Beta Gadget  Advanced beta model with extras

Specific Regex Use Cases

Selecting Rows Containing a Substring Pattern

To find rows where 'Description' contains "model" or "Model":

import pandas as pd
import re

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

regex_contains_model = r'model' # Simple regex, effectively a substring search

df_contains_model = df[df['Description'].str.contains(regex_contains_model, flags=re.IGNORECASE)]
print("Rows where 'Description' contains 'model' (case-insensitive):")
print(df_contains_model)

Output:

Rows where 'Description' contains 'model' (case-insensitive):
      ID Product Name                      Description
1  Y203A  Beta Gadget  Advanced beta model with extras

Selecting Rows Ending With a Substring Pattern

To find rows where 'ID' ends with a letter followed by a number (e.g., 'A1', 'B2'):

import pandas as pd

df = pd.DataFrame({
    'ID': ['X101', 'Y203A', 'X105B', 'Z300', 'Y203C'],
    'Product Name': ['Alpha Widget', 'Beta Gadget', 'alpha gizmo', 'Gamma Unit', 'Delta Device'],
    'Description': ['High-quality alpha series', 'Advanced beta model with extras', 'Compact gizmo for Alphas', 'Standard gamma component', 'Next-gen delta system']
})

regex_ends_letter_num = r'[A-Za-z]\d$' # [A-Za-z] for one letter, \d for one digit, $ for end
df_ends_letter_num = df[df['ID'].str.contains(regex_ends_letter_num)]
print("Rows where 'ID' ends with LetterNumber:")
print(df_ends_letter_num)

Output:

Rows where 'ID' ends with LetterNumber:
Empty DataFrame
Columns: [ID, Product Name, Description]
Index: []

Conclusion

Filtering Pandas DataFrame rows using regular expressions is a powerful technique for selecting data based on complex string patterns.

  • The primary method is df['ColumnName'].str.contains(your_regex_pattern).
  • This returns a boolean Series, which is then used for indexing the DataFrame: df[boolean_mask].
  • Use the case=False parameter or flags=re.IGNORECASE (after import re) within str.contains() for case-insensitive matching.
  • Use the tilde operator (~) to negate the boolean mask and select rows that do not match the regex.
  • For simple "starts with," "ends with," or literal substring checks, Pandas built-in string methods like str.startswith(), str.endswith(), or str.contains(literal_string, regex=False) can be more direct and readable alternatives.

By mastering str.contains() with regex, you gain significant control over how you filter and subset your textual data in Pandas.

Navigation