PyTorch Tensor Shape

View tensor shape

There are two ways to view the tensor shape:

• View Tensor.shape via properties
• View Tensor.size() by method

The result of both methods is an object of type torch.Size (a subclass of tuple)

>>> t = torch.empty(3, 4)
>>>t.size()
torch.Size([3, 4])
# Get the size of dim=1 dimension
>>>t.size(dim=1)
4
>>> t.shape
torch.Size([3, 4])
>>> isinstance(t.shape, tuple)
True

Change tensor shape

1. Return new tensor

The following methods all require: The number of elements before and after the shape change is the same. Methods whose names end with as refer to the shape of other tensors.

• Tensor.view(*shape)
• Tensor.view_as(other)
• Tensor.reshape(*shape)
• Tensor.reshape_as(other)

In the case where the tensor data is contiguous, both methods share the underlying data.

>>> t = torch.arange(6)
>>> t_1 = t.view(2, 3)
>>> t_1
tensor([[0, 1, 2],
        [3, 4, 5]])
# There can be one dimension that is -1, and PyTorch infers the size of the dimension based on the remaining dimensions.
>>> t_2 = t.reshape(3, -1)
>>> t_2
tensor([[0, 1],
        [twenty three],
        [4, 5]])
# Get the underlying data address
>>> p = t.storage().data_ptr()
>>> p_1 = t_1.storage().data_ptr()
>>> p_2 = t_2.storage().data_ptr()
# The underlying data addresses are equal
>>> p == p_1 == p_2
True

When the tensor data is discontinuous (such as changing the dimension order of the tensor), the Tensor.view() method cannot be used, and the Tensor.reshape() method will copy the underlying layer. data

>>> a = torch.tensor([[0, 1],[2, 3]])
>>>a
tensor([[0, 1],
        [twenty three]])
# Transpose operation
>>> t = a.t()
>>> t
tensor([[0, 2],
        [1, 3]])
# t is no longer continuous, of course a is still continuous
>>> t.is_contiguous()
False

# t.view throws exception
>>>t.view(1, 4)
-------------------------------------------------- ----------
RuntimeError Traceback (most recent call last)
Input In [10], in <cell line: 1>()
----> 1 t.view(1, 4)

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

# t.reshape will copy data
>>> t_2 = t.reshape(1, 4)
>>> t_2
tensor([[0, 2, 1, 3]])
>>> t.storage().data_ptr() == t_2.storage().data_ptr()
False

2. Modify tensor in place

This is a relatively low-level method, ending with _ means directly modifying the tensor itself, without creating a new tensor, and does not require the same number of elements before and after.

• Tensor.resize_(*sizes)
• Tensor.resize_as_(other)

Code example:

>>> t = torch.arange(5)
>>> t
tensor([0, 1, 2, 3, 4])
>>> t.resize_(2, 2)
tensor([[0, 1],
        [twenty three]])
# The original tensor has been modified
>>> t
tensor([[0, 1],
        [twenty three]])

Adjust dimension order

1. Matrix transpose

• Tensor.t()

Only valid for 2-dimensional tensors, 0-dimensional and 1-dimensional tensors are returned unchanged

>>> t = torch.tensor([[1, 2, 3]])
>>> t
tensor([[1, 2, 3]])
# or t.T
>>>t.t()
tensor([[1],
        [2],
        [3]])

2. swap two dimensions

• Tensor.transpose(dim0, dim1)

To swap the dim0 and dim1 dimensions of a Tensor, Tensor.t() is equivalent to Tensor.transpose(0, 1)

3. Adjust dimension order

• Tensor.permute(*dims)

Specify the order of Tensor dimensions and convert the image tensor in HxWxC order to CxHxW order. You can call: img.permute(2, 0, 1)

4. Reverse dimension order

• Tensor.T(deprecated)

Equivalent to Tensor.permute(n-1, n-2, …, 0), in future versions Tensor.T will only be valid for two-dimensional tensors, that is, matrix transposition

• Tensor.mT

The last two dimensions represent the matrix, all the previous dimensions represent mini-batch, only the last two dimensions are reversed, and the order of the remaining dimensions remains unchanged.

>>> t = torch.arange(6).view(1,2,3)
>>> t.shape
torch.Size([1, 2, 3])
>>> t.permute(2, 1, 0).shape
torch.Size([3, 2, 1])
>>> t.T.shape
torch.Size([3, 2, 1])

Insert or remove dimensions

• Insert dimension of size 1 Tensor.unsqueeze(dim)

• Delete dimensions of size 1 Tensor.squeeze(dim=None) Delete all 1-dim when dim is None

The dim parameter specifies the index bit for insertion or deletion

>>> t = torch.arange(4).view(2, -1)
>>> t.shape
torch.Size([2, 2])

# Insert 1-dim at the beginning
>>> t_2 = t.unsqueeze(0)
>>> t_2.shape
torch.Size([1, 2, 2])

# Insert 1-dim at the end
>>> t_3 = t_2.unsqueeze(-1)
>>> t_3.shape
torch.Size([1, 2, 2, 1])

# squeeze removes all 1-dim by default
>>> t_3.squeeze().shape
torch.Size([2, 2])

Duplicate dimension data

1. Tensor.expand(*sizes)

The *sizes parameter specifies the size of each dimension after expansion. -1 means that the dimension is not expanded, and only 1-dim data can be expanded.

This method does not allocate additional space, shares the underlying data with the original tensor, and can add dimensions on the left

>>> t = torch.tensor([1, 2]).view(2, 1)
>>> t
tensor([[1],
        [2]])
>>> t.shape
torch.Size([2, 1])

# Expand dimension (2, 1) to (2, 3),
# 2 in the original dimension (2, 1) remains unchanged, and 1 is expanded to 3
>>> e = t.expand(-1, 3)
>>>e
tensor([[1, 1, 1],
        [2, 2, 2]])
>>>e.shape
torch.Size([2, 3])

# Modifying the original tensor will affect the expanded tensor
>>> t[0, 0] = 0
>>>e
tensor([[0, 0, 0],
        [2, 2, 2]])

2. Tensor.repeat(*sizes)

The *sizes parameter specifies the number of repetitions of each dimension. Repeating once is equivalent to the dimensions remaining unchanged.

This method copies the underlying data. Similar to Tensor.expand(), you can also add dimensions on the left

>>> t = torch.tensor([1, 2]).view(2, 1)
>>> t
tensor([[1],
        [2]])
>>> t.shape
torch.Size([2, 1])

# # Both 2 and 1 in the original dimension (2, 1) are repeated 2 times
>>> r = t.repeat(2, 2)
>>> r.shape
torch.Size([4, 2])
>>> r
tensor([[1, 1],
        [twenty two],
        [1, 1],
        [twenty two]])

Splicing

torch.cat(tensors, dim=0) Splice multiple tensors, Except for the splicing dimension specified by the dim parameter, other dimensions must be consistent

>>> a = torch.arange(3, 9).view(3, 2).t()
>>>a
tensor([[3, 5, 7],
        [4, 6, 8]])
>>> b = torch.tensor([[1], [2]])
>>> b
tensor([[1],
        [2]])
>>> b.shape
torch.Size([2, 1])
# Dimension (2, 3) and dimension (2, 1) splicing
# The 1st dimension is the same (both are 2), the 2nd dimension is different (spliced on this dimension)
>>> torch.cat([b, a], dim=1)
tensor([[1, 3, 5, 7],
        [2, 4, 6, 8]])

Overlay

torch.stack(tensors, dim=0) Stack multiple tensors, The dimensions of all tensors must be consistent, dim specifies the new dimension after superposition< /font>

>>> t = torch.arange(4).view(2, 2)
>>> t
tensor([[0, 1],
        [twenty three]])
# After superposition, it becomes a three-dimensional tensor.
>>> n = torch.stack([t, t, t], dim=2)
>>>n
tensor([[[0, 0, 0],
         [1, 1, 1]],

        [[2, 2, 2],
         [3, 3, 3]]])
>>> n.shape
torch.Size([2, 2, 3])

>>> t = torch.arange(10).view(2, 5)
>>> t
tensor([[0, 1, 2, 3, 4],
        [5, 6, 7, 8, 9]])
# Split by columns. Each tensor has 2 columns. There are 5 columns in total. The last tensor is not divided enough and has only 1 column.
>>> t.split(2, dim=1)
(tensor([[0, 1],
         [5, 6]]),
 tensor([[2, 3],
         [7, 8]]),
 tensor([[4],
         [9]]))
# 3 tensors are 1, 3, 1 columns respectively
>>> t.split([1, 3, 1], dim=1)
(tensor([[0],
         [5]]),
 tensor([[1, 2, 3],
         [6, 7, 8]]),
 tensor([[4],
         [9]]))

>>> t = torch.arange(10).view(2, 5)
>>> t
tensor([[0, 1, 2, 3, 4],
        [5, 6, 7, 8, 9]])
# Divide 5 columns into 3 pieces. It cannot be divided equally. The last piece is slightly smaller.
>>> t.chunk(3, dim=1)
(tensor([[0, 1],
         [5, 6]]),
 tensor([[2, 3],
         [7, 8]]),
 tensor([[4],
         [9]]))