Kronecker product

From Wikipedia, the free encyclopedia

In mathematics, the Kronecker product, denoted by $\otimes$ , is an operation on two matrices of arbitrary size resulting in a block matrix. It is a special case of a tensor product. The Kronecker product should not be confused with the usual matrix multiplication, which is an entirely different operation. It is named after German mathematician Leopold Kronecker.

1 Definition
- 1.1 Examples
2 Properties
3 Matrix equations
4 History
5 Related matrix operations
6 References
7 External links

[edit] Definition

If A is an m-by-n matrix and B is a p-by-q matrix, then the Kronecker product $A \otimes B$ is the mp-by-nq block matrix

$A \otimes B = \begin{bmatrix} a_{11} B & \cdots & a_{1n}B \\ \vdots & \ddots & \vdots \\ a_{m1} B & \cdots & a_{mn} B \end{bmatrix}.$

More explicitly, we have

$A \otimes B = \begin{bmatrix} a_{11} b_{11} & a_{11} b_{12} & \cdots & a_{11} b_{1q} & \cdots & \cdots & a_{1n} b_{11} & a_{1n} b_{12} & \cdots & a_{1n} b_{1q} \\ a_{11} b_{21} & a_{11} b_{22} & \cdots & a_{11} b_{2q} & \cdots & \cdots & a_{1n} b_{21} & a_{1n} b_{22} & \cdots & a_{1n} b_{2q} \\ \vdots & \vdots & \ddots & \vdots & & & \vdots & \vdots & \ddots & \vdots \\ a_{11} b_{p1} & a_{11} b_{p2} & \cdots & a_{11} b_{pq} & \cdots & \cdots & a_{1n} b_{p1} & a_{1n} b_{p2} & \cdots & a_{1n} b_{pq} \\ \vdots & \vdots & & \vdots & \ddots & & \vdots & \vdots & & \vdots \\ \vdots & \vdots & & \vdots & & \ddots & \vdots & \vdots & & \vdots \\ a_{m1} b_{11} & a_{m1} b_{12} & \cdots & a_{m1} b_{1q} & \cdots & \cdots & a_{mn} b_{11} & a_{mn} b_{12} & \cdots & a_{mn} b_{1q} \\ a_{m1} b_{21} & a_{m1} b_{22} & \cdots & a_{m1} b_{2q} & \cdots & \cdots & a_{mn} b_{21} & a_{mn} b_{22} & \cdots & a_{mn} b_{2q} \\ \vdots & \vdots & \ddots & \vdots & & & \vdots & \vdots & \ddots & \vdots \\ a_{m1} b_{p1} & a_{m1} b_{p2} & \cdots & a_{m1} b_{pq} & \cdots & \cdots & a_{mn} b_{p1} & a_{mn} b_{p2} & \cdots & a_{mn} b_{pq} \end{bmatrix}.$

[edit] Examples

$\begin{bmatrix} 1 & 2 \\ 3 & 4 \\ \end{bmatrix} \otimes \begin{bmatrix} 0 & 5 \\ 6 & 7 \\ \end{bmatrix} = \begin{bmatrix} 1\cdot 0 & 1\cdot 5 & 2\cdot 0 & 2\cdot 5 \\ 1\cdot 6 & 1\cdot 7 & 2\cdot 6 & 2\cdot 7 \\ 3\cdot 0 & 3\cdot 5 & 4\cdot 0 & 4\cdot 5 \\ 3\cdot 6 & 3\cdot 7 & 4\cdot 6 & 4\cdot 7 \\ \end{bmatrix} = \begin{bmatrix} 0 & 5 & 0 & 10 \\ 6 & 7 & 12 & 14 \\ 0 & 15 & 0 & 20 \\ 18 & 21 & 24 & 28 \end{bmatrix}$ .

$\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \end{bmatrix} \otimes \begin{bmatrix} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \end{bmatrix} = \begin{bmatrix} a_{11} b_{11} & a_{11} b_{12} & a_{11} b_{13} & a_{12} b_{11} & a_{12} b_{12} & a_{12} b_{13} \\ a_{11} b_{21} & a_{11} b_{22} & a_{11} b_{23} & a_{12} b_{21} & a_{12} b_{22} & a_{12} b_{23} \\ a_{21} b_{11} & a_{21} b_{12} & a_{21} b_{13} & a_{22} b_{11} & a_{22} b_{12} & a_{22} b_{13} \\ a_{21} b_{21} & a_{21} b_{22} & a_{21} b_{23} & a_{22} b_{21} & a_{22} b_{22} & a_{22} b_{23} \\ a_{31} b_{11} & a_{31} b_{12} & a_{31} b_{13} & a_{32} b_{11} & a_{32} b_{12} & a_{32} b_{13} \\ a_{31} b_{21} & a_{31} b_{22} & a_{31} b_{23} & a_{32} b_{21} & a_{32} b_{22} & a_{32} b_{23} \end{bmatrix}$ .

[edit] Properties

[edit] Bilinearity and associativity

The Kronecker product is a special case of the tensor product, so it is bilinear and associative:

$A \otimes (B+C) = A \otimes B + A \otimes C \qquad,$

$(A+B) \otimes C = A \otimes C + B \otimes C \qquad,$

$(kA) \otimes B = A \otimes (kB) = k(A \otimes B),$

$(A \otimes B) \otimes C = A \otimes (B \otimes C),$

where A, B and C are matrices and k is a scalar.

The Kronecker product is not commutative: in general, A $\otimes$ B and B $\otimes$ A are different matrices. However, A $\otimes$ B and B $\otimes$ A are permutation equivalent, meaning that there exist permutation matrices P and Q such that

$A \otimes B = P \, (B \otimes A) \, Q.$

If A and B are square matrices, then A $\otimes$ B and B $\otimes$ A are even permutation similar, meaning that we can take P = Q^T.

[edit] The mixed-product property

If A, B, C and D are matrices of such size that one can form the matrix products AC and BD, then

$(A \otimes B)(C \otimes D) = AC \otimes BD.$

This is called the mixed-product property, because it mixes the ordinary matrix product and the Kronecker product. It follows that A $\otimes$ B is invertible if and only if A and B are invertible, in which case the inverse is given by

$(A \otimes B)^{-1} = A^{-1} \otimes B^{-1}.$

[edit] Kronecker sum and exponentiation

If A is n-by-n, B is m-by-m and $I k$ denotes the k-by-k identity matrix then we can define the Kronecker sum, $\oplus$ , by

$A \oplus B = A \otimes I_m + I_n \otimes B.$

We have the following formula for the matrix exponential which is useful in the numerical evaluation of certain continuous-time Markov processes^{[citation needed]},

$e^{A \oplus B} = e^A \otimes e^B.$

[edit] Spectrum

Suppose that A and B are square matrices of size n and q respectively. Let λ₁, ..., λ_n be the eigenvalues of A and μ₁, ..., μ_q be those of B (listed according to multiplicity). Then the eigenvalues of A $\otimes$ B are

$\lambda_i \mu_j, \qquad i=1,\ldots,n ,\, j=1,\ldots,q.$

It follows that the trace and determinant of a Kronecker product are given by

$\operatorname{tr}(A \otimes B) = \operatorname{tr} A \, \operatorname{tr} B \quad\mbox{and}\quad \det(A \otimes B) = (\det A)^q (\det B)^n.$

[edit] Singular values

If A and B are rectangular matrices, then one can consider their singular values. Suppose that A has r_A nonzero singular values, namely

$\sigma_{A,i}, \qquad i = 1, \ldots, r_A.$

Similarly, denote the nonzero singular values of B by

$\sigma_{B,i}, \qquad i = 1, \ldots, r_B.$

Then the Kronecker product A $\otimes$ B has r_Ar_B nonzero singular values, namely

$\sigma_{A,i} \sigma_{B,j}, \qquad i=1,\ldots,r_A ,\, j=1,\ldots,r_B.$

Since the rank of a matrix equals the number of nonzero singular values, we find that

$\operatorname{rank}(A \otimes B) = \operatorname{rank} A \, \operatorname{rank} B.$

[edit] Relation to the abstract tensor product

The Kronecker product of matrices corresponds to the abstract tensor product of linear maps. Specifically, if the matrices A and B represent linear transformations V₁ → W₁ and V₂ → W₂, respectively, then the matrix A $\otimes$ B represents the tensor product of the two maps, V₁ $\otimes$ V₂ → W₁ $\otimes$ W₂.

[edit] Relation to products of graphs

The Kronecker product of the adjacency matrices of two graphs is the adjacency matrix of the tensor product graph. The Kronecker sum of the adjacency matrices of two graphs is the adjacency matrix of the Cartesian product graph. See ^[1], answer to Exercise 96.

[edit] Transpose

The operation of transposition is distributive over the Kronecker product:

$(A\otimes B)^T = A^T \otimes B^T.$

[edit] Matrix equations

The Kronecker product can be used to get a convenient representation for some matrix equations. Consider for instance the equation AXB = C, where A, B and C are given matrices and the matrix X is the unknown. We can rewrite this equation as

$(B^\top \otimes A) \, \operatorname{vec}(X) = \operatorname{vec}(AXB) = \operatorname{vec}(C).$

It now follows from the properties of the Kronecker product that the equation AXB = C has a unique solution if and only if A and B are nonsingular (Horn & Johnson 1991, Lemma 4.3.1).

Here, vec(X) denotes the vectorization of the matrix X formed by stacking the columns of X into a single column vector.

If X is row-ordered into the column vector x then $A X B$ can be also be written as $(A \otimes B^\top)x$ (Jain 1989, 2.8 Block Matrices and Kronecker Products)

[edit] History

The Kronecker product is named after Leopold Kronecker, even though there is little evidence that he was the first to define and use it. Indeed, in the past the Kronecker product was sometimes called the Zehfuss matrix, after Johann Georg Zehfuss.

[edit] Related matrix operations

Two related matrix operations are the Tracy-Singh and Khatri-Rao products which operate on partitioned matrices. Let the $m$ -by- $n$ matrix $A$ be partitioned into the $m i$ -by- $n j$ blocks $A i j$ and $p$ -by- $q$ matrix $B$ into the $p k$ -by- $q l$ blocks B_kl with of course $Σ i m i = m$ , $Σ j n j = n$ , $Σ k p k = p$ and $Σ l q l = q .$

The Tracy-Singh product^[2]^[3] is defined as

$A \circ B = (A_{ij}\circ B)_{ij} = ((A_{ij} \otimes B_{kl})_{kl})_{ij}$

which means that the $(i j)$ th subblock of the $m p$ -by- $n q$ product $A\circ B$ is the $m i p$ -by- $n j q$ matrix $A_{ij} \circ B$ , of which the $(k l)$ th subblock equals the $m i p k$ -by- $n j q l$ matrix $A_{ij} \otimes B_{kl}$ . Essentially the Tracy-Singh product is the pairwise Kronecker product for each pair of partitions in the two matrices.

For example, if $A$ and $B$ both are $2$ -by- $2$ partitioned matrices e.g.:

$A = \left[ \begin{array} {c | c} A_{11} & A_{12} \\ \hline A_{21} & A_{22} \end{array} \right] = \left[ \begin{array} {c c | c} 1 & 2 & 3 \\ 4 & 5 & 6 \\ \hline 7 & 8 & 9 \end{array} \right] ,\quad B = \left[ \begin{array} {c | c} B_{11} & B_{12} \\ \hline B_{21} & B_{22} \end{array} \right] = \left[ \begin{array} {c | c c} 1 & 4 & 7 \\ \hline 2 & 5 & 8 \\ 3 & 6 & 9 \end{array} \right] ,$

we get:

$A \circ B = \left[ \begin{array} {c | c} A_{11} \circ B & A_{12} \circ B \\ \hline A_{21} \circ B & A_{22} \circ B \end{array} \right] = \left[ \begin{array} {c | c | c | c } A_{11} \otimes B_{11} & A_{11} \otimes B_{12} & A_{12} \otimes B_{11} & A_{12} \otimes B_{12} \\ \hline A_{11} \otimes B_{21} & A_{11} \otimes B_{22} & A_{12} \otimes B_{21} & A_{12} \otimes B_{22} \\ \hline A_{21} \otimes B_{11} & A_{21} \otimes B_{12} & A_{22} \otimes B_{11} & A_{22} \otimes B_{12} \\ \hline A_{21} \otimes B_{21} & A_{21} \otimes B_{22} & A_{22} \otimes B_{21} & A_{22} \otimes B_{22} \end{array} \right]$

$= \left[ \begin{array} {c c | c c c c | c | c c} 1 & 2 & 4 & 7 & 8 & 14 & 3 & 12 & 21 \\ 4 & 5 & 16 & 28 & 20 & 35 & 6 & 24 & 42 \\ \hline 2 & 4 & 5 & 8 & 10 & 16 & 6 & 15 & 24 \\ 3 & 6 & 6 & 9 & 12 & 18 & 9 & 18 & 27 \\ 8 & 10 & 20 & 32 & 25 & 40 & 12 & 30 & 48 \\ 12 & 15 & 24 & 36 & 30 & 45 & 18 & 36 & 54 \\ \hline 7 & 8 & 28 & 49 & 32 & 56 & 9 & 36 & 63 \\ \hline 14 & 16 & 35 & 56 & 40 & 64 & 18 & 45 & 72 \\ 21 & 24 & 42 & 63 & 48 & 72 & 27 & 54 & 81 \end{array} \right].$

The Khatri-Rao product^[4]^[5] is defined as

$A \ast B = (A_{ij}\otimes B_{ij})_{ij}$

in which the $(i j)$ th block is the $m i p i$ -by- $n j q j$ sized Kronecker product of the corresponding blocks of $A$ and $B$ , assuming the number of row and column partitions of both matrices is equal. The size of the product is then $Σ i m i p i$ -by- $Σ j n j q j$ . Proceeding with the same matrices as the previous example we obtain:

$A \ast B = \left[ \begin{array} {c | c} A_{11} \otimes B_{11} & A_{12} \otimes B_{12} \\ \hline A_{21} \otimes B_{21} & A_{22} \otimes B_{22} \end{array} \right] = \left[ \begin{array} {c c | c c} 1 & 2 & 12 & 21 \\ 4 & 5 & 24 & 42 \\ \hline 14 & 16 & 45 & 72 \\ 21 & 24 & 54 & 81 \end{array} \right].$

This is a submatrix of the Tracy-Singh product of the two matrices (each partition in this example is a partition in a corner of the Tracy-Singh product).

A column-wise Kronecker product of two matrices may also be called the Khatri-Rao product. This product assumes the partitions of the matrices are their columns. In this case $m 1 = m$ , $p 1 = p$ , $n = q$ and $\forall j: n_j=p_j=1$ . The resulting product is a $m p$ -by- $n$ matrix of which each column is the Kronecker product of the corresponding columns of $A$ and $B$ . Using the matrices from the previous examples with the columns partitioned:

$C = \left[ \begin{array} { c | c | c} C_1 & C_2 & C_3 \end{array} \right] = \left[ \begin{array} {c | c | c} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{array} \right] ,\quad D = \left[ \begin{array} { c | c | c } D_1 & D_2 & D_3 \end{array} \right] = \left[ \begin{array} { c | c | c } 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \end{array} \right] ,$

so that:

$C \ast D = \left[ \begin{array} { c | c | c } C_1 \otimes D_1 & C_2 \otimes D_2 & C_3 \otimes D_3 \end{array} \right] = \left[ \begin{array} { c | c | c } 1 & 8 & 21 \\ 2 & 10 & 24 \\ 3 & 12 & 27 \\ 4 & 20 & 42 \\ 8 & 25 & 48 \\ 12 & 30 & 54 \\ 7 & 32 & 63 \\ 14 & 40 & 72 \\ 21 & 48 & 81 \end{array} \right].$

[edit] References

^ D. E. Knuth: "Pre-Fascicle 0a: Introduction to Combinatorial Algorithms", zeroth printing (revision 2), to appear as part of D.E. Knuth: The Art of Computer Programming Vol. 4A
^ Tracy, DS, Singh RP. 1972. A new matrix product and its applications in matrix differentiation. Statistica Neerlandica 26: 143-157.
^ Liu S. 1999. Matrix results on the Khatri-Rao and Tracy-Singh products. Linear Algebra and its Applications 289: 267-277. (pdf)
^ Khatry CG, Rao CR. 1968. Solutions to some functional equations and their applications to characterization of probability distributions. Sankhya 30: 167-180.
^ Zhang X, Yang Z, Cao C. 2002. Inequalities involving Khatri-Rao products of positive semi-definite matrices. Applied Mathematics E-notes 2: 117-124.

Horn, Roger A. & Johnson, Charles R. (1991), Topics in Matrix Analysis, Cambridge University Press, ISBN 0-521-46713-6 .

Jain, Anil K. (1989), Fundamentals of Digital Image Processing, Prentice Hall, ISBN 0-13-336165-9 .

[edit] External links

Categories: Matrix theory

Hidden categories: All articles with unsourced statements | Articles with unsourced statements since January 2008