Skip to content
Tutorial Reference

Python Pandas: How to Fix "ValueError: Cannot mask with non-boolean array containing NA / NaN values"

Learn how to resolve the Pandas error when filtering DataFrames with boolean masks containing NaN values, especially when using Series.str.contains().

When filtering Pandas DataFrames using boolean masks, particularly those generated by string methods like Series.str.contains(), you might encounter the ValueError: Cannot mask with non-boolean array containing NA / NaN values. This error signals that the boolean Series you're attempting to use for filtering isn't purely boolean; it contains NaN (Not a Number) or None values. Pandas requires a mask to consist entirely of True or False to unambiguously determine which rows to keep.

This guide will clearly explain why NaN values in a boolean mask cause this ValueError, demonstrate scenarios with str.contains() on columns with missing or non-string data, and provide robust solutions, including using the na parameter in str.contains(), explicit fillna(), or ensuring consistent string types.

Understanding the Error: The Requirement for Pure Boolean Masks

In Pandas, when you filter a DataFrame using boolean indexing like df[boolean_mask], the boolean_mask (which is typically a Pandas Series) must contain only boolean values (True or False).

  • Rows where the mask is True are kept.
  • Rows where the mask is False are dropped.

If the boolean_mask contains NaN (Not a Number) or Python's None, Pandas cannot definitively decide whether to keep or drop the corresponding row. NaN doesn't have a clear boolean interpretation in this context, leading to the ValueError.

Common Cause 1: NaN/None Values in the Column Used with str.contains()

The Series.str.contains(pattern) method tests if pattern is found within each string of the Series. How it handles missing values (NaN, None) in the input Series is key.

How str.contains() Handles NaN by Default

By default, if Series.str.contains() encounters a NaN or None value in the input Series, the corresponding value in the output boolean Series will also be NaN (or None if dtype is object, often represented as pd.NA in some contexts for string dtypes).

Let's use a sample DataFrame:

import pandas as pd
import numpy as np # For np.nan

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

print("Original DataFrame:")
print(df)

Output:

Original DataFrame:
      customer_name  order_id
0  Alice Wonderland       101
1     Robert Tables       102
2              None       103
3      Diana Prince       104
4    Charles Xavier       105
5               NaN       106

Reproducing the Error

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

boolean_mask_with_na = df['customer_name'].str.contains('Alice')
print("Boolean mask generated by str.contains('Alice'):")
print(boolean_mask_with_na)

try:
    filtered_df_error = df[boolean_mask_with_na]
    print(filtered_df_error)
except ValueError as e:
    print(f"Error: {e}")

Output:

Boolean mask generated by str.contains('Alice'):
0     True
1    False
2     None
3    False
4    False
5      NaN
Name: customer_name, dtype: object
Error: Cannot mask with non-boolean array containing NA / NaN values

The Series.str.contains() method has an na parameter. Setting na=False tells Pandas to treat missing values in the input Series as if they do not contain the pattern (i.e., return False for them). This produces a purely boolean mask.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

boolean_mask_na_false = df['customer_name'].str.contains('Alice', na=False)
print("Boolean mask with na=False:")
print(boolean_mask_na_false)

filtered_df_na_false = df[boolean_mask_na_false]
print("Filtered DataFrame (using na=False):")
print(filtered_df_na_false)

Output:

Boolean mask with na=False:
0     True
1    False
2    False
3    False
4    False
5    False
Name: customer_name, dtype: bool
Filtered DataFrame (using na=False):
      customer_name  order_id
0  Alice Wonderland       101

This is generally the cleanest and most direct solution when using str.contains().

Solution: Explicitly Compare Result with True

Comparing the result of str.contains() (which might contain None/NaN) with == True will convert None/NaN comparisons to False, effectively creating a pure boolean mask.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

boolean_mask_equals_true = (df['customer_name'].str.contains('Alice') == True)
print("Boolean mask after '== True':")
print(boolean_mask_equals_true)

filtered_df_equals_true = df[boolean_mask_equals_true]
print("Filtered DataFrame (using '== True'):")
print(filtered_df_equals_true)

Output:

Boolean mask after '== True':
0     True
1    False
2    False
3    False
4    False
5    False
Name: customer_name, dtype: bool
Filtered DataFrame (using '== True'):
      customer_name  order_id
0  Alice Wonderland       101

Solution: Use .fillna(False) After str.contains()

You can generate the mask with str.contains() and then explicitly fill any resulting NaN/None values with False.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

boolean_mask_fillna = df['customer_name'].str.contains('Alice').fillna(False)
print("Boolean mask after .fillna(False):")
print(boolean_mask_fillna)
print()

filtered_df_fillna = df[boolean_mask_fillna]
print("Filtered DataFrame (using .fillna(False)):")
print(filtered_df_fillna)

Output:

Boolean mask after .fillna(False):
0     True
1    False
2    False
3    False
4    False
5    False
Name: customer_name, dtype: bool

Filtered DataFrame (using .fillna(False)):
      customer_name  order_id
0  Alice Wonderland       101

Common Cause 2: Non-String Values in the Column Used with str.contains()

The Series.str.contains() method is designed to work on string data. If your column has a mixed data type (e.g., contains numbers or booleans alongside strings), .str.contains() will produce NaN for the non-string entries, leading to the same ValueError when used as a mask.

Reproducing the Error

import pandas as pd

df_mixed_type = pd.DataFrame({
    'product_code': ['A101', 'B202', 303, 'D404', True], # Mixed types
    'quantity': [10, 5, 15, 8, 20]
})

print("DataFrame with mixed types in 'product_code':")
print(df_mixed_type)
print(f"dtype of 'product_code': {df_mixed_type['product_code'].dtype}\n")

boolean_mask_mixed_type = df_mixed_type['product_code'].str.contains('A')
print("Mask from .str.contains('A') on mixed type column:")
print(boolean_mask_mixed_type)

try:
    filtered_df_mixed_error = df_mixed_type[boolean_mask_mixed_type]
except ValueError as e:
    print(f"Error with mixed type column: {e}")

Output:

DataFrame with mixed types in 'product_code':
  product_code  quantity
0         A101        10
1         B202         5
2          303        15
3         D404         8
4         True        20
dtype of 'product_code': object

Mask from .str.contains('A') on mixed type column:
0     True
1    False
2      NaN
3    False
4      NaN
Name: product_code, dtype: object
Error with mixed type column: Cannot mask with non-boolean array containing NA / NaN values

Solution: Convert Column to String Type using .astype(str)

Before applying .str.contains(), convert the entire column to string type using .astype(str). This ensures all values are strings, and str.contains() can operate correctly (though you still need to handle original NaNs if they become the string 'nan').

import pandas as pd

df_mixed_type = pd.DataFrame({
    'product_code': ['A101', 'B202', 303, 'D404', True], # Mixed types
    'quantity': [10, 5, 15, 8, 20]
})


boolean_mask_astype_str = df_mixed_type['product_code'].astype(str).str.contains('A', na=False)
print("Mask after .astype(str).str.contains('A', na=False):")
print(boolean_mask_astype_str)
print()

filtered_df_astype_str = df_mixed_type[boolean_mask_astype_str]
print("Filtered DataFrame after .astype(str):")
print(filtered_df_astype_str)

Output:

Mask after .astype(str).str.contains('A', na=False):
0     True
1    False
2    False
3    False
4    False
Name: product_code, dtype: bool

Filtered DataFrame after .astype(str):
  product_code  quantity
0         A101        10

Note: If your original column had np.nan or None, astype(str) would convert these to the string 'nan' or 'None'. Using na=False in the subsequent str.contains is still a good idea, or fillna('') before astype(str) if you want original NaNs to become empty strings that don't match most patterns.

Alternative (Data-Altering) Solution: Dropping Rows with NaNs using dropna()

If you decide that rows with missing values in the key column are not relevant, you can remove them using DataFrame.dropna(subset=['your_column']) before creating the boolean mask. This ensures the mask is generated from a Series without NaNs. This is a data-altering step.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_name': ['Alice Wonderland', 'Robert Tables', None, 'Diana Prince', 'Charles Xavier', np.nan],
    'order_id': [101, 102, 103, 104, 105, 106]
})

df_dropped_na = df.copy()
df_dropped_na.dropna(subset=['customer_name'], inplace=True)
print("DataFrame after dropping rows with NaN in 'customer_name':")
print(df_dropped_na)
print()

boolean_mask_after_dropna = df_dropped_na['customer_name'].str.contains('Alice')
filtered_df_after_dropna = df_dropped_na[boolean_mask_after_dropna]
print("Filtered DataFrame after dropna():")
print(filtered_df_after_dropna)

Output:

DataFrame after dropping rows with NaN in 'customer_name':
      customer_name  order_id
0  Alice Wonderland       101
1     Robert Tables       102
3      Diana Prince       104
4    Charles Xavier       105

Filtered DataFrame after dropna():
      customer_name  order_id
0  Alice Wonderland       101

Conclusion

The ValueError: Cannot mask with non-boolean array containing NA / NaN values in Pandas arises when your boolean mask intended for filtering contains non-boolean NaN or None values. When this occurs with Series.str.contains():

  1. For columns with NaN/None values: The most direct solution is to use na=False within the Series.str.contains('pattern', na=False) call. Alternatively, chain .fillna(False) after str.contains() or use an equality comparison (Series.str.contains('pattern') == True).
  2. For columns with non-string data types: Convert the column to string using your_series.astype(str) before applying .str.contains(). Remember to also handle potential NaNs that might result from this conversion or were originally present.
  3. Consider dropna() as a pre-processing step if rows with missing values in the target column are to be excluded entirely.

By ensuring your boolean mask is purely True or False, you can perform reliable filtering operations in Pandas.

Navigation