Skip to content
Tutorial Reference

How to Safely Access and Merge Nested Dictionaries in Python

Learn how to safely traverse and recursively merge deeply nested Python dictionaries without KeyError crashes or losing data during updates.

Working with deeply nested dictionaries, common when handling JSON APIs, can be error-prone. One missing key in a chain like data['users'][0]['profile']['location'] causes a crash.

This guide provides professional, robust utilities for safely accessing and recursively merging complex dictionaries without external dependencies.

The Problem with Standard Access

Python's standard dictionary access raises a KeyError if a key is missing. When keys are nested, handling this requires deeply nested try-except blocks or verbose .get() chaining.

try:
    city = data['user']['profile']['address']['city']
except (KeyError, TypeError):
    city = 'Unknown'

city = data.get('user', {}).get('profile', {}).get('address', {}).get('city', 'Unknown')

Both approaches reduce code readability and maintainability.

Solution 1: Safe Access with Dot Notation

Instead of writing repetitive error handling, we can create a utility function that accepts a "dot-path" string (e.g., "user.profile.name"). We use functools.reduce to traverse the dictionary dynamically.

The get_nested Helper

from functools import reduce

def get_nested(data, path, default=None):
    """
    Get value from a nested dictionary using a dot-path string.
    
    Args:
        data (dict): The dictionary to search.
        path (str): The dot-separated path (e.g., 'meta.pagination.page').
        default (any): The value to return if the path doesn't exist.
    """
    try:
        return reduce(lambda d, k: d[k], path.split('.'), data)
    except (KeyError, TypeError):
        return default

Usage Example

response = {
    'user': {
        'id': 101,
        'profile': {
            'name': 'Alice',
            'settings': None # Note: settings is not a dict
        }
    }
}

name = get_nested(response, 'user.profile.name')
print(f"Name: {name}")   # Output: Alice

age = get_nested(response, 'user.profile.age', 'N/A')
print(f"Age: {age}")     # Output: N/A

theme = get_nested(response, 'user.profile.settings.theme', 'Dark Mode')
print(f"Theme: {theme}") # Output: Dark Mode

Output:

Name: Alice
Age: N/A
Theme: Dark Mode

Solution 2: Deep Merging (Recursive Update)

Python's built-in dict.update() method performs a shallow merge. It completely overwrites top-level keys. If you are merging configuration files where you only want to override specific nested fields while keeping others, update() destroys your data.

The Problem with Shallow Update

config = {
    "db": {"host": "localhost", "port": 5432},
    "debug": True
}

overrides = {
    "db": {"host": "prod-db"}
}

config.update(overrides)
print(config)

Output:

{'db': {'host': 'prod-db'}, 'debug': True}

But we lost the port!

The deep_update Helper

To preserve nested data, we need a recursive function that checks if both the source and the override are dictionaries before merging.

def deep_update(source, overrides):
    """
    Recursively update a dictionary.
    
    Args:
        source (dict): The dictionary to be updated in-place.
        overrides (dict): The dictionary containing changes.
    """
    for key, value in overrides.items():
        if isinstance(value, dict) and isinstance(source.get(key), dict):
            deep_update(source[key], value)
        else:
            source[key] = value
    return source

Usage Example

import json

def deep_update(source, overrides):
    """
    Recursively update a dictionary.
    
    Args:
        source (dict): The dictionary to be updated in-place.
        overrides (dict): The dictionary containing changes.
    """
    for key, value in overrides.items():
        if isinstance(value, dict) and isinstance(source.get(key), dict):
            deep_update(source[key], value)
        else:
            source[key] = value
    return source

config = {
    "db": {"host": "localhost", "port": 5432},
    "logging": {"level": "INFO", "file": "app.log"}
}

env_overrides = {
    "db": {"host": "prod-db"},       # Only change host, keep port
    "logging": {"level": "ERROR"}    # Only change level, keep file
}

deep_update(config, env_overrides)

print(json.dumps(config, indent=2))

Output:

{
  "db": {
    "host": "prod-db",
    "port": 5432
  },
  "logging": {
    "level": "ERROR",
    "file": "app.log"
  }
}

Production Considerations

While these helper functions are excellent for scripts and lightweight applications, robust production systems often use specialized libraries for data validation and schema definition.

  1. Pydantic: The industry standard for data validation. It allows you to define classes (Models) for your data. If data is missing or wrong, it raises precise errors.
  2. Glom: A library specifically designed for restructuring and accessing nested data in Python.

However, for simple configuration merging or API response handling, get_nested and deep_update are lightweight, dependency-free solutions.

Summary

Operation Standard Method Recommendation
Deep Access d['a']['b']['c'] Use get_nested(d, 'a.b.c') to avoid crashes.
Updating d.update(new) Use deep_update(d, new) to preserve nested keys.
Navigation