Skip to content
Tutorial Reference

Python Pandas: How to Replace None (and 'None' Strings) with NaN

Learn how to convert Python None and string 'None' values to np.nan in Pandas using fillna() and replace() for consistent missing data handling in DataFrames.

In data analysis with Pandas, missing data is often represented by None (Python's null object) or sometimes as the literal string "None". For numerical computations and consistent missing data handling within Pandas, it's standard practice to convert these to numpy.nan (Not a Number), which is Pandas' canonical representation for missing floating-point data.

This guide explains how to use DataFrame.fillna() and DataFrame.replace() to effectively replace None values and "None" strings with NaN in your DataFrames.

Understanding None vs. NaN in Pandas

  • None: Python's built-in null object. When a column in Pandas has mixed types and contains None, its dtype is often object.
  • numpy.nan (NaN): Stands for "Not a Number." It's a special floating-point value used by Pandas (and NumPy) to represent missing numerical data. Columns containing NaN (and otherwise numbers) will typically have a float dtype.
  • Why Convert? Using NaN allows for consistent missing data handling across Pandas and NumPy, enabling vectorized numerical operations to correctly skip or propagate missing values. Many Pandas methods (like .isnull(), .dropna(), .sum()) are designed to work seamlessly with NaN.

Example DataFrame:

import pandas as pd
import numpy as np # For np.nan

data = {
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],             # Contains Python None
    'Score': [85, 90, None, 77, 88],                                # Contains Python None
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']   # Contains "None" string
}
df_original = pd.DataFrame(data)
print("Original DataFrame:")
print(df_original)
print()

print("Original dtypes:")
print(df_original.dtypes)

Output:

Original DataFrame:
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102     None   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0      None
4  105      Eve   88.0   Pending

Original dtypes:
ID          int64
Name       object
Score     float64
Status     object
dtype: object

The DataFrame.fillna(value) method is specifically designed to fill missing values (which includes None and NaN by default).

Replacing in the Entire DataFrame

To replace all occurrences of None (and existing NaNs) with np.nan across the entire DataFrame:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']
})
df_filled = df.fillna(value=np.nan) # This effectively ensures all missing are np.nan

print("DataFrame after df.fillna(np.nan):")
print(df_filled)
print()

print("Dtypes after df.fillna(np.nan):")
print(df_filled.dtypes)

Output:

DataFrame after df.fillna(np.nan):
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102      NaN   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0      None
4  105      Eve   88.0   Pending

Dtypes after df.fillna(np.nan):
ID          int64
Name       object
Score     float64
Status     object
dtype: object

Replacing in a Specific Column

To target a specific column:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']
})
df_col_filled = df.copy()

df_col_filled['Name'] = df_col_filled['Name'].fillna(value=np.nan)

print("DataFrame after filling 'Name' column:")
print(df_col_filled)

Output:

DataFrame after filling 'Name' column:
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102      NaN   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0      None
4  105      Eve   88.0   Pending

Method 2: Replacing None and/or "None" Strings with NaN using DataFrame.replace()

The DataFrame.replace(to_replace, value) method is more general and can replace any specified value(s) with another value.

Replacing None Values

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']
})

df_replaced_none = df.replace(to_replace=[None], value=np.nan)

print("DataFrame after df.replace(None, np.nan):")
print(df_replaced_none)

Output: (Similar to fillna, all Python None objects become np.nan)

DataFrame after df.replace(None, np.nan):
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102      NaN   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0      None
4  105      Eve   88.0   Pending

Replacing "None" Strings

If your DataFrame contains the literal string "None" representing missing data:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending'] # Has "None" string
})
df_replaced_str_none = df.replace(to_replace="None", value=np.nan)

print("DataFrame after df.replace('None', np.nan):")
print(df_replaced_str_none)

Output:

DataFrame after df.replace('None', np.nan):
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102     None   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0       NaN
4  105      Eve   88.0   Pending

Replacing Both None Values and "None" Strings

Provide a list to to_replace to handle multiple types of missing value representations.

import pandas as pd
import numpy as np

data_mixed_missing = {
    'Name': ['Alice', None, 'Charlie', 'None', 'David'], # Python None and "None" string
    'Age': [25, 30, None, 22, 'None'] # Python None and "None" string, and numbers
}
df_mixed = pd.DataFrame(data_mixed_missing)
print("Original mixed missing DataFrame:")
print(df_mixed)
print()

df_replaced_both = df_mixed.replace(to_replace=[None, "None"], value=np.nan)

print("DataFrame after replacing both None and 'None' string:")
print(df_replaced_both)

Output:

Original mixed missing DataFrame:
      Name   Age
0    Alice    25
1     None    30
2  Charlie  None
3     None    22
4    David  None

DataFrame after replacing both None and 'None' string:
      Name   Age
0    Alice  25.0
1      NaN  30.0
2  Charlie   NaN
3      NaN  22.0
4    David   Na

Replacing in Specific Columns

You can call .replace() on a specific column (Series) or a selection of columns.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']
})
df_col_replace = df.copy()

df_col_replace['Status'] = df_col_replace['Status'].replace(to_replace="None", value=np.nan)
print("DataFrame after replacing 'None' string in 'Status' column only:")
print(df_col_replace)

Output:

DataFrame after replacing 'None' string in 'Status' column only:
    ID     Name  Score    Status
0  101    Alice   85.0    Active
1  102     None   90.0  Inactive
2  103  Charlie    NaN    Active
3  104    David   77.0       NaN
4  105      Eve   88.0   Pending

Caution with replace() and Datetime Columns

If you use df.replace(to_replace=[None], value=np.nan) on a DataFrame that includes datetime columns (or columns that should be datetime but have None and are thus object type), the None values will become NaT (Not a Time, Pandas' missing value for datetimes) if the column is already datetime type. However, if an object column containing None and strings is broadly replaced, None becomes np.nan (a float), which can prevent subsequent conversion to datetime if not handled. It's often better to use fillna() on datetime columns or convert them to datetime after general None to NaN replacements on other columns.

For object columns that you intend to be datetime, but have Nones:

df_dt = pd.DataFrame({'event_date': ['2023-01-01', None, '2023-03-15']})
df_dt['event_date'] = pd.to_datetime(df_dt['event_date']) # This converts None to NaT correctly
print("Datetime column with NaT:")
print(df_dt)
print(df_dt.dtypes)

Making Changes In-Place (inplace=True)

Both fillna() and replace() return a new DataFrame by default. To modify the original DataFrame directly, use the inplace=True argument.

import pandas as pd
import numpy as np

df_inplace_example = pd.DataFrame({
    'ID': [101, 102, 103, 104, 105],
    'Name': ['Alice', None, 'Charlie', 'David', 'Eve'],
    'Score': [85, 90, None, 77, 88],
    'Status': ['Active', 'Inactive', 'Active', 'None', 'Pending']
})


print("Before inplace replace (Name has None):")
print(df_inplace_example['Name'])
print()

df_inplace_example['Name'].replace(to_replace=None, value=np.nan, inplace=True)

print("After inplace replace (Name has NaN):")
print(df_inplace_example['Name'])
print()

Conclusion

To standardize missing values in a Pandas DataFrame by converting None or "None" strings to numpy.nan:

  • For replacing Python's None objects, df.fillna(value=np.nan) is generally the most idiomatic and direct method.
  • For replacing literal strings like "None" (or a list of multiple representations of missing data including Python's None), df.replace(to_replace=["None", None], value=np.nan) is more flexible.
  • These methods can be applied to the entire DataFrame or specific columns.
  • Remember that introducing np.nan into an integer column will convert that column's dtype to float.
  • Be mindful when applying broad replacements to DataFrames with datetime-like columns; handle them specifically or convert to datetime type first using pd.to_datetime(), which correctly handles None by converting to NaT.

By using these methods, you can ensure consistent representation of missing data in your Pandas DataFrames, facilitating more robust data analysis and processing.

Navigation