Python PyTorch: How to Perform Element-Wise Addition on Tensors

Element-wise addition is one of the most fundamental operations in deep learning and numerical computing. When working with PyTorch tensors, you frequently need to add corresponding elements of two tensors together - whether for combining feature maps in neural networks, applying residual connections, or performing basic mathematical transformations.

In this guide, you'll learn how to perform element-wise addition on PyTorch tensors using torch.add() and the + operator, handle tensors of different dimensions through broadcasting, and add scalar values to tensors.

Understanding torch.add()

PyTorch provides the torch.add() function for element-wise addition:

torch.add(input, other, *, alpha=1, out=None)

Parameters:

Returns: A new tensor containing the element-wise sum.

Adding Two 1D Tensors

The simplest case is adding two tensors of the same shape. Each element in the first tensor is added to the corresponding element in the second:

import torch

tens_1 = torch.Tensor([10, 20, 30, 40, 50])
tens_2 = torch.Tensor([1, 2, 3, 4, 5])

result = torch.add(tens_1, tens_2)
print("Tensor 1:", tens_1)
print("Tensor 2:", tens_2)
print("Result:  ", result)

Output:

Tensor 1: tensor([10., 20., 30., 40., 50.])
Tensor 2: tensor([1., 2., 3., 4., 5.])
Result:   tensor([11., 22., 33., 44., 55.])

You can also use the + operator, which produces identical results:

result = tens_1 + tens_2
print("Result:", result)

Output:

Result: tensor([11., 22., 33., 44., 55.])

Adding Two 2D Tensors

Element-wise addition works the same way with multi-dimensional tensors - each element at position [i][j] in the first tensor is added to the element at the same position in the second:

import torch

tens_1 = torch.Tensor([[1, 2], [3, 4]])
tens_2 = torch.Tensor([[10, 20], [30, 40]])

result = torch.add(tens_1, tens_2)

print("Tensor 1:")
print(tens_1)
print("\nTensor 2:")
print(tens_2)
print("\nResult:")
print(result)

Output:

Tensor 1:
tensor([[1., 2.],
        [3., 4.]])

Tensor 2:
tensor([[10., 20.],
        [30., 40.]])

Result:
tensor([[11., 22.],
        [33., 44.]])

Adding Tensors of Different Dimensions (Broadcasting)

PyTorch supports broadcasting, which allows element-wise operations on tensors with different shapes. When dimensions don't match, PyTorch automatically expands the smaller tensor to match the larger one, following specific broadcasting rules.

import torch

tens_1 = torch.Tensor([1, 2])

tens_2 = torch.Tensor([[10, 20], [30, 40]])

result = torch.add(tens_1, tens_2)

print("Tensor 1 (1D):", tens_1)
print("Tensor 1 shape:", tens_1.shape)
print("\nTensor 2 (2D):")
print(tens_2)
print("Tensor 2 shape:", tens_2.shape)
print("\nResult:")
print(result)
print("Result shape:", result.shape)

Output:

Tensor 1 (1D): tensor([1., 2.])
Tensor 1 shape: torch.Size([2])

Tensor 2 (2D):
tensor([[10., 20.],
        [30., 40.]])
Tensor 2 shape: torch.Size([2, 2])

Result:
tensor([[11., 22.],
        [31., 42.]])
Result shape: torch.Size([2, 2])

Here, the 1D tensor [1, 2] is broadcast (repeated) across each row of the 2D tensor before addition. The result always has the shape of the higher-dimensional tensor.

Adding a Scalar to a Tensor

You can add a single number (scalar) to every element of a tensor:

import torch

tens_1d = torch.Tensor([1, 2, 3, 4])
tens_2d = torch.Tensor([[10, 20], [30, 40]])

result_1d = torch.add(tens_1d, 10)
result_2d = torch.add(tens_2d, 20)

print("1D tensor + 10:", result_1d)
print("2D tensor + 20:")
print(result_2d)

Output:

1D tensor + 10: tensor([11., 12., 13., 14.])
2D tensor + 20:
tensor([[30., 40.],
        [50., 60.]])

The scalar is broadcast to match the tensor's shape, effectively adding it to every element.

Using the alpha Parameter for Scaled Addition

The alpha parameter multiplies the second tensor before adding it. This computes input + alpha * other, which is useful for operations like weighted sums and gradient updates:

import torch

tens_1 = torch.Tensor([10, 20, 30])
tens_2 = torch.Tensor([1, 2, 3])

result = torch.add(tens_1, tens_2, alpha=5)
print("Result (input + 5 * other):", result)

Output:

Result (input + 5 * other): tensor([15., 30., 45.])

This is equivalent to tens_1 + 5 * tens_2 but can be more efficient since it avoids creating an intermediate tensor.

In-Place Addition with add_()

If you want to modify a tensor in place (without creating a new tensor), use the add_() method. In-place operations in PyTorch are denoted by a trailing underscore:

import torch

tens = torch.Tensor([1, 2, 3, 4])
print("Before:", tens)

tens.add_(10)
print("After in-place addition:", tens)

Output:

Before: tensor([1., 2., 3., 4.])
After in-place addition: tensor([11., 12., 13., 14.])

Common Mistake: Shape Mismatch

Element-wise addition requires tensors to have compatible shapes. If the shapes can't be broadcast, PyTorch raises an error:

import torch

tens_1 = torch.Tensor([1, 2, 3])       # Shape: (3,)
tens_2 = torch.Tensor([10, 20, 30, 40]) # Shape: (4,)

try:
    result = torch.add(tens_1, tens_2)
except RuntimeError as e:
    print(f"Error: {e}")

Output:

Error: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 0

Fix: Ensure your tensors have matching or broadcastable shapes before performing addition. Reshape or pad tensors as needed:

import torch

tens_1 = torch.Tensor([1, 2, 3])
tens_2 = torch.Tensor([10, 20, 30, 40])

result = torch.add(tens_1, tens_2[:3])
print("Sliced result:", result)

tens_1_padded = torch.nn.functional.pad(tens_1, (0, 1), value=0)
result = torch.add(tens_1_padded, tens_2)
print("Padded result:", result)

Output:

Sliced result: tensor([11., 22., 33.])
Padded result: tensor([11., 22., 33., 40.])

torch.add() vs + Operator

Use torch.add() when you need the alpha or out parameters. Use + for simple, readable addition.

Summary

Element-wise addition in PyTorch is straightforward with either torch.add() or the + operator.

Both support tensors of any dimension and leverage broadcasting to handle tensors of different shapes automatically.

Use the alpha parameter for scaled addition, avoid in-place operations (add_()) on tensors that require gradients, and always verify that your tensor shapes are compatible to prevent runtime errors.