Cholesky decomposition

In linear algebra, the Cholesky decomposition or Cholesky factorization is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose, which is useful e.g. for efficient numerical solutions and Monte Carlo simulations. It was discovered by André-Louis Cholesky for real matrices. When it is applicable, the Cholesky decomposition is roughly twice as efficient as the LU decomposition for solving systems of linear equations.[1]

Statement

The Cholesky decomposition of a Hermitian positive-definite matrix A is a decomposition of the form

\mathbf{A = L L}^{*}

where L is a lower triangular matrix with real and positive diagonal entries, and L* denotes the conjugate transpose of L. Every Hermitian positive-definite matrix (and thus also every real-valued symmetric positive-definite matrix) has a unique Cholesky decomposition.[2]

If the matrix A is Hermitian and positive semi-definite, then it still has a decomposition of the form A = LL* if the diagonal entries of L are allowed to be zero.[3]

When A has real entries, L has real entries as well and the factorization may be written A = LLT [4]

The Cholesky decomposition is unique when A is positive definite; there is only one lower triangular matrix L with strictly positive diagonal entries such that A = LL*. However, the decomposition need not be unique when A is positive semidefinite.

The converse holds trivially: if A can be written as LL* for some invertible L, lower triangular or otherwise, then A is Hermitian and positive definite.

LDL decomposition

A closely related variant of the classical Cholesky decomposition is the LDL decomposition,

\mathbf{A = L D L}^{*}

where L is a lower unit triangular (unitriangular) matrix and D is a diagonal matrix.

This decomposition is related to the classical Cholesky decomposition, of the form LL*, as follows:

\mathbf{A = L D L}^{*} = \mathbf L \mathbf D^{\frac 1 2} \mathbf D^{{\frac 1 2}{*}} \mathbf L^{*} = \mathbf L \mathbf D^{\frac 1 2} (\mathbf L \mathbf D^{\frac 1 2})^{*}

The LDL variant, if efficiently implemented, requires the same space and computational complexity to construct and use but avoids extracting square roots.[5] Some indefinite matrices for which no Cholesky decomposition exists have an LDL decomposition with negative entries in D. For these reasons, the LDL decomposition may be preferred. For real matrices, the factorization has the form A = LDLT and is often referred to as LDLT decomposition (or LDLT decomposition). It is closely related to the eigendecomposition of real symmetric matrices, A=QΛQT.

Example

Here is the Cholesky decomposition of a symmetric real matrix:

\begin{align} \left( \begin{array}{*{3}{r}} 4 & 12 & -16 \\ 12 & 37 & -43 \\ -16 & -43 & 98 \\ \end{array} \right) & = \left( \begin{array}{*{3}{r}} 2 & 0 & 0 \\ 6 & 1 & 0 \\ -8 & 5 & 3 \\ \end{array} \right) \left( \begin{array}{*{3}{r}} 2 & 6 & -8 \\ 0 & 1 & 5 \\ 0 & 0 & 3 \\ \end{array} \right) \end{align}

And here is its LDLT decomposition:

\begin{align} \left( \begin{array}{*{3}{r}} 4 & 12 & -16 \\ 12 & 37 & -43 \\ -16 & -43 & 98 \\ \end{array} \right) & = \left( \begin{array}{*{3}{r}} 1 & 0 & 0 \\ 3 & 1 & 0 \\ -4 & 5 & 1 \\ \end{array} \right) \left( \begin{array}{*{3}{r}} 4 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 9 \\ \end{array} \right) \left( \begin{array}{*{3}{r}} 1 & 3 & -4 \\ 0 & 1 & 5 \\ 0 & 0 & 1 \\ \end{array} \right) \end{align}

Applications

The Cholesky decomposition is mainly used for the numerical solution of linear equations Ax = b. If A is symmetric and positive definite, then we can solve Ax = b by first computing the Cholesky decomposition A = LL*, then solving Ly = b for y by forward substitution, and finally solving L*x = y for x by back substitution.

For linear systems that can be put into symmetric form, the Cholesky decomposition (or its LDL variant) is the method of choice, for superior efficiency and numerical stability. Compared to the LU decomposition, it is roughly twice as efficient.

Linear least squares

Systems of the form Ax = b with A symmetric and positive definite arise quite often in applications. For instance, the normal equations in linear least squares problems are of this form. It may also happen that matrix A comes from an energy functional which must be positive from physical considerations; this happens frequently in the numerical solution of partial differential equations.

Non-linear optimization

Non-linear multi-variate functions may be minimized over their parameters using variants of Newton's method called quasi-Newton methods. At each iteration, the search takes a step s defined by solving Hs = -g for s, where s is the step, g is the gradient vector of the function's partial first derivatives with respect to the parameters, and H is an approximation to the Hessian matrix of partial second derivatives formed by repeated rank 1 updates at each iteration. Two well-known update formulae are called Davidon–Fletcher–Powell (DFP) and Broyden–Fletcher–Goldfarb–Shanno (BFGS). Loss of the positive-definite condition through round-off error is avoided if rather than updating an approximation to the inverse of the Hessian, one updates the Cholesky decomposition of an approximation of the Hessian matrix itself.

Monte Carlo simulation

The Cholesky decomposition is commonly used in the Monte Carlo method for simulating systems with multiple correlated variables: The correlation matrix is decomposed, to give the lower-triangular L. Applying this to a vector of uncorrelated samples, u, produces a sample vector Lu with the covariance properties of the system being modeled.[6]

For a simplified example that shows the economy one gets from Cholesky's decomposition, say one needs to generate two correlated normal variables x_1 and x_2. All one needs to do is to generate two uncorrelated Gaussian random variables z_1 and z_2. We set x_1 = z_1 and x_2 = \rho z_1 + \sqrt{1-\rho^2} z_2.

Kalman filters

Unscented Kalman filters commonly use the Cholesky decomposition to choose a set of so-called sigma points. The Kalman filter tracks the average state of a system as a vector x of length N and covariance as an N-by-N matrix P. The matrix P is always positive semi-definite, and can be decomposed into LLT. The columns of L can be added and subtracted from the mean x to form a set of 2N vectors called sigma points. These sigma points completely capture the mean and covariance of the system state.

Matrix inversion

The explicit inverse of a Hermitian matrix can be computed via Cholesky decomposition, in a manner similar to solving linear systems, using \textstyle{n^3} operations (\textstyle\frac{1}{2}n^3 multiplications).[5] The entire inversion can even be efficiently performed in-place.

A non-Hermitian matrix B can also be inverted using the following identity, where BB* will always be Hermitian:

\mathbf{B}^{-1} = \mathbf{B}^{*} \mathbf{(B B ^ {*})}^{-1}.

Computation

There are various methods for calculating the Cholesky decomposition. The computational complexity of commonly used algorithms is O(n3) in general. The algorithms described below all involve about n3/3 FLOPs, where n is the size of the matrix A. Hence, they are half the cost of the LU decomposition, which uses 2n3/3 FLOPs (see Trefethen and Bau 1997).

Which of the algorithms below is faster depends on the details of the implementation. Generally, the first algorithm will be slightly slower because it accesses the data in a less regular manner.

The Cholesky algorithm

The Cholesky algorithm, used to calculate the decomposition matrix L, is a modified version of Gaussian elimination.

The recursive algorithm starts with i := 1 and

A(1) := A.

At step i, the matrix A(i) has the following form:

\mathbf{A}^{(i)}= \begin{pmatrix} \mathbf{I}_{i-1} & 0 & 0 \\ 0 & a_{i,i} & \mathbf{b}_{i}^{*} \\ 0 & \mathbf{b}_{i} & \mathbf{B}^{(i)} \end{pmatrix},

where Ii−1 denotes the identity matrix of dimension i − 1.

If we now define the matrix Li by

\mathbf{L}_{i}:= \begin{pmatrix} \mathbf{I}_{i-1} & 0 & 0 \\ 0 & \sqrt{a_{i,i}} & 0 \\ 0 & \frac{1}{\sqrt{a_{i,i}}} \mathbf{b}_{i} & \mathbf{I}_{n-i} \end{pmatrix},

then we can write A(i) as

\mathbf{A}^{(i)} = \mathbf{L}_{i} \mathbf{A}^{(i+1)} \mathbf{L}_{i}^{*}

where

\mathbf{A}^{(i+1)}= \begin{pmatrix} \mathbf{I}_{i-1} & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & \mathbf{B}^{(i)} - \frac{1}{a_{i,i}} \mathbf{b}_{i} \mathbf{b}_{i}^{*} \end{pmatrix}.

Note that bi b*i is an outer product, therefore this algorithm is called the outer product version in (Golub & Van Loan).

We repeat this for i from 1 to n. After n steps, we get A(n+1) = I. Hence, the lower triangular matrix L we are looking for is calculated as

\mathbf{L} := \mathbf{L}_{1} \mathbf{L}_{2} \dots \mathbf{L}_{n}.

The Cholesky–Banachiewicz and Cholesky–Crout algorithms

Access pattern (white) and writing pattern (yellow) for the in-place Cholesky—Banachiewicz algorithm on a 5x5 matrix.

If we write out the equation A = LL*,

\begin{align} {\mathbf{A=LL^T}} & = \begin{pmatrix} L_{11} & 0 & 0 \\ L_{21} & L_{22} & 0 \\ L_{31} & L_{32} & L_{33}\\ \end{pmatrix} \begin{pmatrix} L_{11} & L_{21} & L_{31} \\ 0 & L_{22} & L_{32} \\ 0 & 0 & L_{33} \end{pmatrix} \\ & = \begin{pmatrix} L_{11}^2 & &(\text{symmetric}) \\ L_{21}L_{11} & L_{21}^2 + L_{22}^2& \\ L_{31}L_{11} & L_{31}L_{21}+L_{32}L_{22} & L_{31}^2 + L_{32}^2+L_{33}^2 \end{pmatrix} \end{align}

we obtain the following formula for the entries of L:

L_{j,j} = \sqrt{ A_{j,j} - \sum_{k=1}^{j-1} L_{j,k}^2 }.
L_{i,j} = \frac{1}{L_{j,j}} \left( A_{i,j} - \sum_{k=1}^{j-1} L_{i,k} L_{j,k} \right), \qquad\text{for } i>j.

The expression under the square root is always positive if A is real and positive-definite.

For complex Hermitian matrix, the following formula applies:

L_{j,j} = \sqrt{ A_{j,j} - \sum_{k=1}^{j-1} L_{j,k}L_{j,k}^* }.
L_{i,j} = \frac{1}{L_{j,j}} \left( A_{i,j} - \sum_{k=1}^{j-1} L_{i,k} L_{j,k}^* \right), \qquad\text{for } i>j.

So we can compute the (i, j) entry if we know the entries to the left and above. The computation is usually arranged in either of the following orders.

Either pattern of access allows the entire computation to be performed in-place if desired.

Stability of the computation

Suppose that we want to solve a well-conditioned system of linear equations. If the LU decomposition is used, then the algorithm is unstable unless we use some sort of pivoting strategy. In the latter case, the error depends on the so-called growth factor of the matrix, which is usually (but not always) small.

Now, suppose that the Cholesky decomposition is applicable. As mentioned above, the algorithm will be twice as fast. Furthermore, no pivoting is necessary and the error will always be small. Specifically, if we want to solve Ax = b, and y denotes the computed solution, then y solves the disturbed system (A + E)y = b where

\|\mathbf{E}\|_2 \le c_n \varepsilon \|\mathbf{A}\|_2.

Here, || ||2 is the matrix 2-norm, cn is a small constant depending on n, and ε denotes the unit round-off.

One concern with the Cholesky decomposition to be aware of is the use of square roots. If the matrix being factorized is positive definite as required, the numbers under the square roots are always positive in exact arithmetic. Unfortunately, the numbers can become negative because of round-off errors, in which case the algorithm cannot continue. However, this can only happen if the matrix is very ill-conditioned. One way to address this is to add a diagonal correction matrix to the matrix being decomposed in an attempt to promote the positive-definiteness.[7] While this might lessen the accuracy of the decomposition, it can be very favorable for other reasons; for example, when performing Newton's method in optimization, adding a diagonal matrix can improve stability when far from the optimum.

LDL decomposition

An alternative form, eliminating the need to take square roots, is the symmetric indefinite factorization[8]

\begin{align} {\mathbf{A=LDL}^\mathrm{T}} & = \begin{pmatrix} 1 & 0 & 0 \\ L_{21} & 1 & 0 \\ L_{31} & L_{32} & 1\\ \end{pmatrix} \begin{pmatrix} D_1 & 0 & 0 \\ 0 & D_2 & 0 \\ 0 & 0 & D_3\\ \end{pmatrix} \begin{pmatrix} 1 & L_{21} & L_{31} \\ 0 & 1 & L_{32} \\ 0 & 0 & 1\\ \end{pmatrix} \\ & = \begin{pmatrix} D_1 & &(\mathrm{symmetric}) \\ L_{21}D_1 & L_{21}^2D_1 + D_2& \\ L_{31}D_1 & L_{31}L_{21}D_{1}+L_{32}D_2 & L_{31}^2D_1 + L_{32}^2D_2+D_3. \end{pmatrix} \end{align}

If A is real, the following recursive relations apply for the entries of D and L:

D_j = A_{jj} - \sum_{k=1}^{j-1} L_{jk}^2 D_k
L_{ij} = \frac{1}{D_j} \left( A_{ij} - \sum_{k=1}^{j-1} L_{ik} L_{jk} D_k \right), \qquad\text{for } i>j.

For complex Hermitian matrix A, the following formula applies:

D_{j} = A_{jj} - \sum_{k=1}^{j-1} L_{jk}L_{jk}^* D_k
L_{ij} = \frac{1}{D_j} \left( A_{ij} - \sum_{k=1}^{j-1} L_{ik} L_{jk}^* D_k \right), \qquad\text{for } i>j.

Again, the pattern of access allows the entire computation to be performed in-place if desired.

Block variant

When used on indefinite matrices, the LDL* factorization is known to be unstable without careful pivoting;[9] specifically, the elements of the factorization can grow arbitrarily. A possible improvement is to perform the factorization on block sub-matrices, commonly 2x2:[10]

\begin{align} {\mathbf{A=LDL}^\mathrm{T}} & = \begin{pmatrix} \mathbf I & 0 & 0 \\ \mathbf L_{21} & \mathbf I & 0 \\ \mathbf L_{31} & \mathbf L_{32} & \mathbf I\\ \end{pmatrix} \begin{pmatrix} \mathbf D_1 & 0 & 0 \\ 0 & \mathbf D_2 & 0 \\ 0 & 0 & \mathbf D_3\\ \end{pmatrix} \begin{pmatrix} \mathbf I & \mathbf L_{21}^\mathrm T & \mathbf L_{31}^\mathrm T \\ 0 & \mathbf I & \mathbf L_{32}^\mathrm T \\ 0 & 0 & \mathbf I\\ \end{pmatrix} \\ & = \begin{pmatrix} \mathbf D_1 & &(\mathrm{symmetric}) \\ \mathbf L_{21} \mathbf D_1 & \mathbf L_{21} \mathbf D_1 \mathbf L_{21}^\mathrm T + \mathbf D_2& \\ \mathbf L_{31} \mathbf D_1 & \mathbf L_{31} \mathbf D_{1} \mathbf L_{21}^\mathrm T + \mathbf L_{32} \mathbf D_2 & \mathbf L_{31} \mathbf D_1 \mathbf L_{31}^\mathrm T + \mathbf L_{32} \mathbf D_2 \mathbf L_{32}^\mathrm T + \mathbf D_3 \end{pmatrix} \end{align}

where every element in the matrices above is a square submatrix. From this, these analogous recursive relations follow:

\mathbf D_j = \mathbf A_{jj} - \sum_{k=1}^{j-1} \mathbf L_{jk} \mathbf D_k \mathbf L_{jk}^\mathrm T
\mathbf L_{ij} = \left(\mathbf A_{ij} - \sum_{k=1}^{j-1} \mathbf L_{ik} \mathbf D_k \mathbf L_{jk}^\mathrm T\right) \mathbf D_j^{-1}

Note the presence of matrix products and explicit inversion, this limits the practical block size.

Updating the decomposition

A task that often arises in practice is that one needs to update a Cholesky decomposition. In more details, one has already computed the Cholesky decomposition A = LL* of some matrix A, then one changes the matrix A in some way into another matrix, say \tilde{\mathbf{A}} , and one wants to compute the Cholesky decomposition of the updated matrix: \tilde{\mathbf{A}} = \tilde{\mathbf{L}} \tilde{\mathbf{L}}^* . The question is now whether one can use the Cholesky decomposition of A that was computed before to compute the Cholesky decomposition of \tilde{\mathbf{A}} .

Rank-one update

The specific case, where the updated matrix \tilde{\mathbf{A}} is related to the matrix A by \tilde{\mathbf{A}} = \mathbf{A} + \mathbf{x} \mathbf{x}^* , is known as a rank-one update.

Here is a little function based on [11] written in Matlab syntax which realizes a rank-one update:

function [L] = cholupdate(L,x)
    n = length(x);
    for k=1:n
        r = sqrt(L(k,k)^2 + x(k)^2);
        c = r / L(k, k);
        s = x(k) / L(k, k);
        L(k, k) = r;
        L(k+1:n,k) = (L(k+1:n,k) + s*x(k+1:n)) / c;
        x(k+1:n) = c*x(k+1:n) - s*L(k+1:n,k);
    end
end

Rank-one downdate

A rank-one downdate is similar to a rank-one update, except that the addition is replaced by subtraction: \tilde{\mathbf{A}} = \mathbf{A} - \mathbf{x} \mathbf{x}^* . This only works if the new matrix \tilde{\mathbf{A}} is still positive definite.

The code for the rank-one update shown above can easily be adapted to do a rank-one downdate: one merely needs to replace the two additions in the assignment to r and L(k+1:p,k) by subtractions.

Proof for positive semi-definite matrices

The above algorithms show that every positive definite matrix A has a Cholesky decomposition. This result can be extended to the positive semi-definite case by a limiting argument. The argument is not fully constructive, i.e., it gives no explicit numerical algorithms for computing Cholesky factors.

If A is an n-by-n positive semi-definite matrix, then the sequence {Ak} = {A + (1/k)In} consists of positive definite matrices. (This is an immediate consequence of, for example, the spectral mapping theorem for the polynomial functional calculus.) Also,

Ak → A

in operator norm. From the positive definite case, each Ak has Cholesky decomposition Ak = LkL*k. By property of the operator norm,

\| \mathbf{L}_k \|^2 \le \| \mathbf{L}_k \mathbf{L}_k ^* \| = \| \mathbf{A}_k \|.

So {Lk} is a bounded set in the Banach space of operators, therefore relatively compact (because the underlying vector space is finite-dimensional). Consequently, it has a convergent subsequence, also denoted by {Lk}, with limit L. It can be easily checked that this L has the desired properties, i.e. A = LL* and L is lower triangular with non-negative diagonal entries: for all x and y,

\langle \mathbf{A} x, y \rangle = \langle \lim \mathbf{A}_k x, y \rangle = \langle \lim \mathbf{L}_k \mathbf{L}_k^* x, y \rangle = \langle \mathbf{L} \mathbf{L}^*x, y \rangle.

Therefore, A = LL*. Because the underlying vector space is finite-dimensional, all topologies on the space of operators are equivalent. So Lk tends to L in norm means Lk tends to L entrywise. This in turn implies that, since each Lk is lower triangular with non-negative diagonal entries, L is also.

Generalization

The Cholesky factorization can be generalized to (not necessarily finite) matrices with operator entries. Let \{\mathcal{H}_n \} be a sequence of Hilbert spaces. Consider the operator matrix

\mathbf{A} = \begin{bmatrix} \mathbf{A}_{11} & \mathbf{A}_{12} & \mathbf{A}_{13} & \; \\ \mathbf{A}_{12}^* & \mathbf{A}_{22} & \mathbf{A}_{23} & \; \\ \mathbf{A} _{13}^* & \mathbf{A}_{23}^* & \mathbf{A}_{33} & \; \\ \; & \; & \; & \ddots \end{bmatrix}

acting on the direct sum

\mathcal{H} = \oplus _n \mathcal{H}_n,

where each

\mathbf{A}_{ij} : \mathcal{H}_j \rightarrow \mathcal{H} _i

is a bounded operator. If A is positive (semidefinite) in the sense that for all finite k and for any

h \in \oplus _{n = 1 }^k \mathcal{H}_k ,

we have \langle h, \mathbf{A} h\rangle \ge 0, then there exists a lower triangular operator matrix L such that A = LL*. One can also take the diagonal entries of L to be positive.

Implementations in programming languages

See also

Notes

  1. ↑ Press, William H.; Saul A. Teukolsky; William T. Vetterling; Brian P. Flannery (1992). Numerical Recipes in C: The Art of Scientific Computing (second edition). Cambridge University England EPress. p. 994. ISBN 0-521-43108-5.
  2. ↑ Golub & Van Loan (1996, p. 143), Horn & Johnson (1985, p. 407), Trefethen & Bau (1997, p. 174)
  3. ↑ Golub & Van Loan (1996, p. 147)
  4. ↑ Horn & Johnson (1985, p. 407)
  5. 1 2 Krishnamoorthy, Aravindh; Menon, Deepak (2011). "Matrix Inversion Using Cholesky Decomposition" 1111: 4144. arXiv:1111.4144. Bibcode:2011arXiv1111.4144K.
  6. ↑ Matlab randn documentation.
  7. ↑ Fang, Haw-ren; O’Leary, Dianne P. (8 August 2006). "Modified Cholesky Algorithms: A Catalog with New Approaches" (PDF).
  8. ↑ Watkins, D. (1991). Fundamentals of Matrix Computations. New York: Wiley. p. 84. ISBN 0-471-61414-9.
  9. ↑ Nocedal, Jorge (2000). Numerical Optimization. Springer.
  10. ↑ Fang, Haw-ren (24 August 2007). "Analysis of Block LDLT Factorizations for Symmetric Indefinite Matrices".
  11. ↑ Stewart, G. W. (1998). Basic decompositions. Philadelphia: Soc. for Industrial and Applied Mathematics. ISBN 0-89871-414-1.

References

External links

History of science

Information

Computer code

Use of the matrix in simulation

Online calculators

This article is issued from Wikipedia - version of the Sunday, February 07, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.