Tensor product

The tensor product, denoted by the symbol ⊗, refers to a number of closely related operations in mathematics. The unifying concept is that the tensor product represents the most general bilinear (or multilinear) product. The study of tensors, tensor spaces, and tensor products forms a branch of mathematics called multilinear algebra.

Overview

The tensor product can be thought of as the “most general” bilinear product on a pair of vectors. Given vector spaces V, W, and X, a bilinear map is a function on the cartesian product V × W to X which is separately linear in each argument. That is, fixing either the first or second argument results in a linear map.

The tensor product on vectors is defined to be a bilinear map on V and W to a new vector space V⊗W called the tensor product space:

\otimes \colon V\times W\to V\otimes W.

The elements of V⊗W can be thought of as formal linear combinations of symbols v⊗w for v ∈ V and w ∈ W, subject only to the relations necessary for bilinearity. That is, one must have:

$(v_{1}+v_{2})\otimes w=v_{1}\otimes w+v_{2}\otimes w$
$v\otimes (w_{1}+w_{2})=v\otimes w_{1}+v\otimes w_{2}$
$k(v\otimes w)=(kv)\otimes w=v\otimes (kw)$

Combining the first two properties we have

$(v_{1}+v_{2})\otimes (w_{1}+w_{2})=v_{1}\otimes w_{1}+v_{1}\otimes w_{2}+v_{2}\otimes w_{1}+v_{2}\otimes w_{2}.$

Note that the symbol ⊗ is overloaded: it is used to denote the tensor product of two vectors as well as the tensor product of two vector spaces.

Given a fixed vector space V, one can take the tensor product of V with itself k times:

V^{\otimes k}=V\otimes V\otimes \cdots \otimes V.

The resulting space is called the k^th tensor power of V. The elements of such a space are called tensors of rank k on V. The tensor product of a rank m tensor and a rank n tensor is naturally a rank m+n tensor. By taking the direct sum of all tensor powers of V one obtains a space called the tensor algebra of V. The tensor product (of tensors) is the natural multiplication operator in this algebra.

Tensor product of vector spaces

Let V and W be vector spaces over a fixed field K. The tensor product of V and W is another vector space over K together with a universal bilinear map

\otimes \colon V\times W\to V\otimes W.

The universality of this map means the following: for any vector space X, and any bilinear map B: V×W → X there exists a unique linear map L: V⊗W → X such that L(v⊗w) = B(x, y).

The tensor product space V⊗W is determined up to a unique isomorphism by the above universal property.

Universal property of the tensor product space

Given bases $\{v_{i}\}$ and $\{w_{i}\}$ for V and W respectively, the tensors of the form $v_{i}\otimes w_{j}$ forms a basis for $V\otimes W$ . The dimension of the tensor product therefore is the product of dimensions of the original spaces; for instance $\mathbb {R} ^{m}\otimes \mathbb {R} ^{n}$ will have dimension $mn$ .

Generalizations

See tensor product of modules over a ring

Tensor product of tensors

Let V be a vector space over a field K. For any nonnegative integer k, the k^th tensor power of V is the tensor product of V with itself k times:

T^{k}V=V^{\otimes k}=V\otimes V\otimes \cdots \otimes V.

By convention T⁰V is the ground field K (as a one-dimensional vector space over itself). The elements of T^kV are called tensors of rank k on V.

Associativity of the tensor product means that there is a canonical isomorphism

T^{k}V\otimes T^{\ell }V\cong T^{k+\ell }V.

In other words, given a rank k tensor and a rank ℓ tensor their tensor product can be interpreted as a rank k + ℓ tensor.

Coordinate description

Given a basis {e_i} for V, a basis for T^kV is given by

e_{i_{1}i_{2}\cdots i_{k}}=e_{i_{1}}\otimes e_{i_{2}}\otimes \cdots \otimes e_{i_{k}}

A general rank k tensor is written (using the summation convention) as:

X=X^{i_{1}i_{2}\cdots i_{k}}e_{i_{1}i_{2}\cdots i_{k}}

In terms of these coordinates the tensor product is just given by ordinary multiplication of components. That is,

(X\otimes Y)^{i_{1}i_{2}\cdots i_{k+\ell }}=X^{i_{1}i_{2}\cdots i_{k}}Y^{i_{k+1}i_{k+2}\cdots i_{k+\ell }}.

Tensor product of linear transformations

The set of all linear transformations from one vector space to another forms a vector space itself. One can therefore take the tensor product of linear transformations. Such a tensor product can be naturally interpreted as a linear transformation on a tensor product.

Specifically, there exists is a natural linear map

L(V,V')\otimes L(W,W')\to L(V\otimes W,V'\otimes W')

determined by

(f\otimes g)(v\otimes w)=f(v)\otimes g(w).

This map is a linear embedding, which is to say that its kernel is zero. If all spaces in question are finite-dimensional, this map is a actually a linear isomorphism.

There are several special cases of this embedding which can be obtained by taking one or more of the spaces to be the ground field K. Specifically, one has natural embeddings

V^{*}\otimes W\to L(V,W)\qquad (f\otimes w)(v)=f(v)w

and

V^{*}\otimes W^{*}\to (V\otimes W)^{*}\qquad (f\otimes g)(v\otimes w)=f(v)g(w)

Kronecker product

Main article: Kronecker product.

With matrices this operation is usually called the Kronecker product, a term used to make clear that the result has a particular block structure imposed upon it, in which each element of the first matrix is replaced by the second matrix, scaled by that element. For matrices $U$ and $V$ this is:

U\otimes V={\begin{bmatrix}u_{11}V&u_{12}V&\cdots \\u_{21}V&u_{22}V\\\vdots &&\ddots \end{bmatrix}}

Relation with the dual space

Note that the space $(V\otimes W)^{\star }$ (the dual space of $V\otimes W$ containing all linear functionals on that space) corresponds naturally to the space of all bilinear functionals on $V\times W$ . In other words, every bilinear functional is a functional on the tensor product, and vice versa. There is a natural isomorphism between $V^{\star }\otimes W^{\star }$ and $(V\otimes W)^{\star }$ . So, the tensors of the linear functionals are bilinear functionals. This gives us a new way to look at the space of bilinear functionals, as a tensor product itself.

Tensor product of multilinear maps

Given multilinear maps $f(x_{1},...x_{k})$ and $g(x_{1},...x_{m})$ their tensor product is the multilinear function

(f\otimes g)(x_{1},...x_{k+m})=f(x_{1},...x_{k})g(x_{k+1},...x_{k+m})

Computer programming

Tensors in computer programming are almost always represented as n-dimensional arrays of numbers, where n is the rank of the tensor. The tensor product is an operation which takes a rank-n tensor and a rank-m tensor and computes the resulting rank-(n+m) tensor.

A prototypical algorithm for doing this is given below in the C programming language. Here a is a rank-two tensor and b is a rank-one tensor, with indices i, j, and k, respectively.

  for( int i = 0; i < i_dim; i++)
     for( int j = 0; j < j_dim; j++)
        for( int k = 0; k < k_dim; k++)
           result[i][j][k] = a[i][j]*b[k];

Many array programming languages may have this pattern built in. For example, in APL the tensor product is expressed as A ∘.× B, while in J the tensor product is the expressed as a */ b. In Mathematica, the tensor product is given by

Outer[Times, a, b]

However, these kinds of notation are not universally present in all array languages. Some languages may require explicit treatment of indices (for example, Matlab).

Another interesting example of the tensor product comes from SQL. Here the tensor product of a and b can be given by the following code:

select 
    a.i as i,
    a.j as j,
    b.k as k,
    a.value * b.value as value
  from
    a
    outer join b

User:Fropuff/Drafts/Tensor product