Matrix norm

In mathematics, a matrix norm is a natural extension of the notion of a vector norm to matrices.

Definition

In what follows, $K$ will denote the field of real or complex numbers. Let $K^{m \times n}$ denote the vector space containing all matrices with $m$ rows and $n$ columns with entries in $K$ . Throughout the article $A^*$ denotes the conjugate transpose of matrix $A$ .

A matrix norm is a vector norm on $K^{m \times n}$ . That is, if $\|A\|$ denotes the norm of the matrix $A$ , then,

$\|A\|\ge 0$
$\|A\|= 0$ iff $A=0$
$\|\alpha A\|=|\alpha| \|A\|$ for all $\alpha$ in $K$ and all matrices $A$ in $K^{m \times n}$
$\|A+B\| \le \|A\|+\|B\|$ for all matrices $A$ and $B$ in $K^{m \times n}.$

Additionally, in the case of square matrices (thus, m = n), some (but not all) matrix norms satisfy the following condition, which is related to the fact that matrices are more than just vectors:

$\|AB\| \le \|A\|\|B\|$ for all matrices $A$ and $B$ in $K^{n \times n}.$

A matrix norm that satisfies this additional property is called a sub-multiplicative norm (in some books, the terminology matrix norm is used only for those norms which are sub-multiplicative). The set of all n-by-n matrices, together with such a sub-multiplicative norm, is an example of a Banach algebra.

Induced norm

Main article: Operator norm

If vector norms on K^m and Kⁿ are given (K is field of real or complex numbers), then one defines the corresponding induced norm or operator norm on the space of m-by-n matrices as the following maxima:

\begin{align} \|A\| &= \sup\{\|Ax\| : x\in K^n \mbox{ with }\|x\|= 1\} \\ &= \sup\left\{\frac{\|Ax\|}{\|x\|} : x\in K^n \mbox{ with }x\ne 0\right\}. \end{align}

The operator norm corresponding to the p-norm for vectors is:

\left \| A \right \| _p = \sup \limits _{x \ne 0} \frac{\left \| A x\right \| _p}{\left \| x\right \| _p}.

These are different from the entrywise p-norms and the Schatten p-norms for matrices treated below, which are also usually denoted by $\left \| A \right \| _p .$

In the case of $p=1$ and $p=\infty$ , the norms can be computed as:

\left \| A \right \| _1 = \max \limits _{1 \leq j \leq n} \sum _{i=1} ^m | a_{ij} |,

which is simply the maximum absolute column sum of the matrix.

\left \| A \right \| _\infty = \max \limits _{1 \leq i \leq m} \sum _{j=1} ^n | a_{ij} |,

which is simply the maximum absolute row sum of the matrix

For example, if the matrix A is defined by

A = \begin{bmatrix} -3 & 5 & 7 \\ 2 & 6 & 4 \\ 0 & 2 & 8 \\ \end{bmatrix},

then we have ||A||₁ = max(|-3|+2+0, 5+6+2, 7+4+8) = max(5,13,19) = 19. and ||A||_∞ = max(|-3|+5+7, 2+6+4,0+2+8) = max(15,12,10) = 15.

In the special case of p = 2 (the Euclidean norm), the induced matrix norm is the spectral norm. The spectral norm of a matrix A is the largest singular value of A i.e. the square root of the largest eigenvalue of the positive-semidefinite matrix A^*A:

\left \| A \right \| _2=\sqrt{\lambda_{\text{max}}(A^{^*} A)}=\sigma_{\text{max}}(A)

^[1]

where A^* denotes the conjugate transpose of A.

More generally, one can define the subordinate matrix norm on $K^{m\times n}$ induced by $\|\cdot\|_{\alpha}$ on $K^n$ , and $\|\cdot\|_{\beta}$ on $K^m$ as:

\left \| A \right \| _{\alpha,\beta} = \max \limits _{x \ne 0} \frac{\left \| A x\right \|_{\beta}}{\left \| x\right \|_{\alpha}}.

Subordinate norms are consistent with the norms that induce them, giving

\|Ax\|_{\beta}\leq \|A\|_{\alpha,\beta}\|x\|_{\alpha}.

Any induced operator norm is a sub-multiplicative matrix norm since $\| ABx \| \leq \| A \| \| Bx \| \leq \| A \| \| B \| \| x \|$ and $\max \limits _{ \| x \| = 1} \| AB x \| = \| AB\| .$ .

Any induced norm satisfies the inequality

\left \| A^r \right \|^{1/r} \ge \rho(A),

where ρ(A) is the spectral radius of A. For a symmetric or hermitian matrix $A$ , we have equality for the 2-norm, since in this case the 2-norm is the spectral radius of $A$ . For an arbitrary matrix, we may not have equality for any norm. Take

A = \begin{bmatrix} 0 & 1 \\ 0 & 0 \\ \end{bmatrix},

the spectral radius of $A$ is 0, but $A$ is not the zero matrix, and so none of the induced norms are equal to the spectral radius of $A$ .

Furthermore, for square matrices we have the spectral radius formula:

\lim_{r\rarr\infty}\|A^r\|^{1/r}=\rho(A).

"Entrywise" norms

These vector norms treat an $m \times n$ matrix as a vector of size $m n$ , and use one of the familiar vector norms.

For example, using the p-norm for vectors, we get:

\Vert A \Vert_{p} = \Vert \mathrm{vec}(A) \Vert_{p} = \left( \sum_{i=1}^m \sum_{j=1}^n |a_{ij}|^p \right)^{1/p}

This is a different norm from the induced p-norm (see above) and the Schatten p-norm (see below), but the notation is the same.

The special case p = 2 is the Frobenius norm, and p = ∞ yields the maximum norm.

L2,1 norm

Let $(a_1, a_2, \cdots, a_n)$ be the columns of matrix $A$ . The $L_{2,1}$ norm ^[2] is a sum of Euclidean norm of columns:

\Vert A \Vert_{2,1} = \sum_{j=1}^n \Vert a_{j} \Vert_2 = \sum_{j=1}^n \left( \sum_{i=1}^m |a_{ij}|^2 \right)^{1/2}

Note here the two indexes $i,j$ of $A_{i,j}$ are treated differently; all matrix norms introduced prior to the L2,1 norm treat the two indexes symmetrically. L2,1 norm is widely used in robust data analysis and sparse coding for feature selection.

L2,1 norm is later generalized into $L_{p,q}$ norm

\Vert A \Vert_{p,q} = \left[\sum_{j=1}^n \left( \sum_{i=1}^m |a_{ij}|^p \right)^{q/p}\right]^{1/q}

Frobenius norm

Main article: Hilbert–Schmidt operator

For p = q = 2, this is called the Frobenius norm or the Hilbert–Schmidt norm, though the latter term is often reserved for operators on Hilbert space. This norm can be defined in various ways:

\|A\|_F=\sqrt{\sum_{i=1}^m\sum_{j=1}^n |a_{ij}|^2}=\sqrt{\operatorname{trace}(A^{{}^*}A)}=\sqrt{\sum_{i=1}^{\min\{m,\,n\}} \sigma_{i}^2}

where A^* denotes the conjugate transpose of A, σ_i are the singular values of A, and the trace function is used. The Frobenius norm is similar to the Euclidean norm on Kⁿ and comes from the Frobenius inner product on the space of all matrices.

The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. This norm is often easier to compute than induced norms and has the useful property of being invariant under rotations. This property follows easily from the trace definition restricted to real matrices,

\|A\|_F^{2} = \|P^{\rm T} B P\|_F^{2} = \operatorname{trace}\left( \left( P^{\rm T} B P \right)^{\rm T} \left( P^{\rm T} B P \right) \right) = \operatorname{trace}(B^{\rm T} B) = \|B\|_F^{2}

where we have used the orthogonal nature of P, $P^{\rm T} P = \mathbf{I}$ and the cyclic nature of the trace, $\operatorname{trace}(XYZ) = \operatorname{trace}(ZXY)$ . More generally the norm is invariant under a unitary transformation for complex matrices.

Max norm

The max norm is the elementwise norm with p = ∞:

\|A\|_{\text{max}} = \max \{|a_{ij}|\}.

This norm is not sub-multiplicative.

Schatten norms

For more details on this topic, see Schatten norm.

The Schatten p-norms arise when applying the p-norm to the vector of singular values of a matrix. If the singular values are denoted by σ_i, then the Schatten p-norm is defined by

\|A\|_p = \left( \sum_{i=1}^{\min\{m,\,n\}} \sigma_i^p \right)^{1/p}. \,

These norms again share the notation with the induced and entrywise p-norms, but they are different.

All Schatten norms are sub-multiplicative. They are also unitarily invariant, which means that ||A|| = ||UAV|| for all matrices A and all unitary matrices U and V.

The most familiar cases are p = 1, 2, ∞. The case p = 2 yields the Frobenius norm, introduced before. The case p = ∞ yields the spectral norm, which is the matrix norm induced by the vector 2-norm (see above). Finally, p = 1 yields the nuclear norm (also known as the trace norm, or the Ky Fan 'n'-norm), defined as

\|A\|_{*} = \operatorname{trace} \left(\sqrt{A^*A}\right) = \sum_{i=1}^{\min\{m,\,n\}} \sigma_i.

(Here $\sqrt{A^*A}$ denotes a positive semidefinite matrix $B$ such that $BB=A^*A$ . More precisely, since $A^*A$ is a positive semidefinite matrix, its square root is well-defined.)

Consistent norms

A matrix norm $\| \cdot \|_{ab}$ on $K^{m \times n}$ is called consistent with a vector norm $\| \cdot \|_{a}$ on $K^n$ and a vector norm $\| \cdot \|_{b}$ on $K^m$ if:

\|Ax\|_b \leq \|A\|_{ab} \|x\|_a

for all $A \in K^{m \times n}, x \in K^n$ . All induced norms are consistent by definition.

Compatible norms

A matrix norm $\| \cdot \|_{b}$ on $K^{n \times n}$ is called compatible with a vector norm $\| \cdot \|_{a}$ on $K^n$ if:

\|Ax\|_a \leq \|A\|_b \|x\|_a

for all $A \in K^{n \times n}, x \in K^n$ . Induced norms are compatible by definition.

Equivalence of norms

For any two vector norms $\|\cdot\|_{\alpha}$ and $\|\cdot\|_{\beta}$ , we have

r\left\|A\right\|_\alpha\leq\left\|A\right\|_\beta\leq s\left\|A\right\|_\alpha

for some positive numbers r and s, for all matrices A in $K^{m \times n}$ . In other words, all norms on $K^{m \times n}$ are equivalent; they induce the same topology on $K^{m \times n}$ . This is true because the vector space $K^{m \times n}$ has the finite dimension $m \times n$ .

Moreover, for every vector norm $\|\cdot\|$ on $\mathbb{R}^{n\times n}$ , there exists a unique positive real number $k$ such that $l\|\cdot\|$ is a sub-multiplicative matrix norm for every $l \ge k$ .

A sub-multiplicative matrix norm $\|\cdot\|_{\alpha}$ is said to be minimal if there exists no other sub-multiplicative matrix norm $\|\cdot\|_{\beta}$ satisfying $\|\cdot\|_{\beta} < \|\cdot\|_{\alpha}$ .

Examples of norm equivalence

For matrix $A\in\mathbb{R}^{m\times n}$ of rank $r$ , the following inequalities hold:^[3]^[4]

$\|A\|_2\le\|A\|_F\le\sqrt{r}\|A\|_2$
$\|A\|_F \le \|A\|_{*} \le \sqrt{r} \|A\|_F$
$\|A\|_{\text{max}} \le \|A\|_2 \le \sqrt{mn}\|A\|_{\text{max}}$
$\frac{1}{\sqrt{n}}\|A\|_\infty\le\|A\|_2\le\sqrt{m}\|A\|_\infty$
$\frac{1}{\sqrt{m}}\|A\|_1\le\|A\|_2\le\sqrt{n}\|A\|_1.$

Here, $\|A\|_p$ refers to the matrix norm induced by the vector p-norm.

Another useful inequality between matrix norms is

\|A\|_2\le\sqrt{\|A\|_1\|A\|_\infty},

which is a special case of Hölder's inequality.

Notes

↑ Carl D. Meyer, Matrix Analysis and Applied Linear Algebra,section 5.2,p281, Society for Industrial & Applied Mathematics,June 2000.
↑ C. Ding, D. Zhou, X. He, H. Zha, "R1-PCA: Rotational Invariant L1-norm Principal Component Analysis for Robust Subspace Factorization", Proc. of Int'l Conf. Machine Learning (ICML 2006), June 2006.
↑ Golub, Gene; Charles F. Van Loan (1996). Matrix Computations - Third Edition. Baltimore: The Johns Hopkins University Press, 56-57. ISBN 0-8018-5413-X.
↑ Roger Horn and Charles Johnson. Matrix Analysis, Chapter 5, Cambridge University Press, 1985. ISBN 0-521-38632-2.

References

James W. Demmel, Applied Numerical Linear Algebra, section 1.7, published by SIAM, 1997.
Carl D. Meyer, Matrix Analysis and Applied Linear Algebra, published by SIAM, 2000.
John Watrous, Theory of Quantum Information, 2.3 Norms of operators, lecture notes, University of Waterloo, 2011.
Kendall Atkinson, An Introduction to Numerical Analysis, published by John Wiley & Sons, Inc 1989