Summary of the product operations(e.g., dot product, matrix product, product with batch) with NumPy and PyTorch.

1. Matrix

1.1 Element-wise product

\[(m, n) \odot (m, n) = (m, n)\]

Initial data:

a = torch.Tensor([[1, 1, 1], 
                  [2, 2, 2]])
b = torch.Tensor([[3, 3, 3], 
                  [0, 0, 0]])
c = torch.Tensor([[2, 2, 2], 
                  [2, 2, 2], 
                  [2, 2, 2]])
print(a.shape)
print(b.shape)
print(c.shape)

torch.Size([2, 3])
torch.Size([2, 3])
torch.Size([3, 3])

Method 1:

a * b

tensor([[3., 3., 3.],
        [0., 0., 0.]])

Method 2:

torch.mul(a, b)

tensor([[3., 3., 3.],
        [0., 0., 0.]])

Method 3:

a.numpy() * b.numpy()

array([[3., 3., 3.],
       [0., 0., 0.]], dtype=float32)

Method 4:

np.multiply(a.numpy(), b.numpy())

array([[3., 3., 3.],
       [0., 0., 0.]], dtype=float32)

1.2 Matrix product

\[(m, n) \circ (n, k) = (m, k)\]

Initial data:

print(a.shape)
print(b.shape)
print(c.shape)

torch.Size([2, 3])
torch.Size([2, 3])
torch.Size([3, 3])

Method 1:

x = a.numpy().dot(c.numpy())
print(x)
print(x.shape)

[[ 6.  6.  6.]
 [12. 12. 12.]]
(2, 3)

Method 2:

x = np.matmul(a.numpy(), c.numpy())
print(x)
print(x.shape)

[[ 6.  6.  6.]
 [12. 12. 12.]]
(2, 3)

Method 3:

x = torch.mm(a, c)
print(x)
print(x.shape)

tensor([[ 6.,  6.,  6.],
        [12., 12., 12.]])
torch.Size([2, 3])

Method 4:

x = torch.matmul(a, c)
print(x)
print(x.shape)

tensor([[ 6.,  6.,  6.],
        [12., 12., 12.]])
torch.Size([2, 3])

with Python 3.5+

Method 5:

a @ c

tensor([[ 6.,  6.,  6.],
        [12., 12., 12.]])

a.numpy() @ c.numpy()

array([[ 6.,  6.,  6.],
       [12., 12., 12.]], dtype=float32)

2. Tensor

2.1 Element-wise product

\[(m, n, k) \odot (m, n, k) = (m, n, k)\]

Initial data:

a = torch.randn(32, 3, 3)
b = torch.randn(3, 5)
c = torch.randn(32, 3, 6)
print(a.shape)
print(b.shape)
print(c.shape)

torch.Size([32, 3, 3])
torch.Size([3, 5])
torch.Size([32, 3, 6])

Method 1:

(a * a).shape

torch.Size([32, 3, 3])

Method 2:

torch.mul(a, a).shape

torch.Size([32, 3, 3])

Method 3:

(a.numpy() * a.numpy()).shape

(32, 3, 3)

Method 4:

np.multiply(a.numpy(), a.numpy()).shape

(32, 3, 3)

2.2 Tensor product with batch I

\[(b, m, n) \circ (n, k) = (b, m, k)\]

Method 1:

a.numpy().dot(b.numpy()).shape

(32, 3, 5)

Method 2:

torch.matmul(a, b).shape

torch.Size([32, 3, 5])

with Python 3.5+

Method 3:

(a @ b).shape

torch.Size([32, 3, 5])

2.3 Tensor product with batch II

\[(b, m, n) \circ (b, n, k) = (b, m, k)\]

Method 1:

np.matmul(a.numpy(), c.numpy()).shape

(32, 3, 6)

Method 2:

torch.matmul(a, c).shape

torch.Size([32, 3, 6])

Method 3:

torch.bmm(a, c).shape

torch.Size([32, 3, 6])

with Python 3.5+

Method 4:

(a @ c).shape

torch.Size([32, 3, 6])