Reproducing kernel Hilbert space

From Wikipedia, the free encyclopedia

In functional analysis (a branch of mathematics), a reproducing kernel Hilbert space is a Hilbert space of functions in which pointwise evaluation is a continuous linear functional. Equivalently, they are spaces that can be defined by reproducing kernels. The subject was originally and simultaneously developed by Nachman Aronszajn (1907-1980) and Stefan Bergman (1895-1987) in 1950.

In this article we assume that Hilbert spaces are complex. The main reason for this is that many of the examples of reproducing kernel Hilbert spaces are spaces of analytic functions, although some real Hilbert spaces also have reproducing kernels.

An important subset of the reproducing kernel Hilbert spaces are the reproducing kernel Hilbert spaces associated to a continuous kernel. These spaces have applications in machine learning.

Contents

[edit] Definition

Let X be an arbitrary set and H a Hilbert space of complex-valued functions on X. We say that H is a reproducing kernel Hilbert space if the linear map

f \mapsto f(x)

from H to the complex numbers is continuous for any x in X. By the Riesz representation theorem, this implies that for every x in X there exists an element Kx of H with the property that:

f(x) = \langle f,\ K_x \rangle \quad \forall f \in H \quad (*)

The function Kx is called the point-evaluation functional at the point x.

Since H is a space of functions, the element Kx is itself a function and can therefore be evaluated at every point. We define the function K: X \times X \to \mathbb{C} by

K(x,y) \ \stackrel{\mathrm{def}}{=}\ K_x(y).

This function is called the reproducing kernel for the Hilbert space H and it is determined entirely by H because the Riesz representation theorem guarantees, for every x in X, that the element Kx satisfying (*) is unique.

[edit] Examples

For example, when X is finite and H consists of all complex-valued functions on X, then an element of H can be represented as an array of complex numbers. If the usual inner product is used, then Kx is the function whose value is 1 at x and 0 everywhere else. In this case, H is isomorphic to \mathbb{C}^n.

[edit] Properties

[edit] The reproducing property

It holds that

\langle K(x,\cdot), K(y,\cdot) \rangle = K(x,y)

[edit] Orthonormal sequences

If \textstyle \left\{ \phi_{k}\right\} _{k=1}^{\infty} is an orthonormal sequence such that the closure of its span is equal to H, then

K\left( x,y\right) =\sum_{k=1}^{\infty}\phi_{k}\left( x\right) \phi _{k}\left( y\right) .

[edit] Moore-Aronszajn theorem

In the previous section, we defined a kernel function in terms of a reproducing kernel Hilbert space. It follows from the definition of an inner product that the kernel we defined is symmetric and positive definite. The Moore-Aronszajn theorem goes in the other direction; it says that every symmetric, positive definite kernel defines a unique reproducing kernel Hilbert space. The theorem first appeared in Aronszajn's Theory of Reproducing Kernels, although he attributes it to E. H. Moore.

Theorem. Suppose K is a symmetric, positive definite kernel on a set E. Then there is a unique Hilbert space of functions on E for which K is a reproducing kernel.

Proof. Define, for all x in E, K_x = K(x, \cdot). Let H0 be the linear span of \{K_x:\ x \in E \} . Define an inner product on H0 by

\left \langle \sum_{j=1}^n b_j K_{y_j}, \sum_{i=1}^m a_i K_{x_i} \right \rangle = \sum_{i=1}^m \sum_{j=1}^n \overline{a_i} b_j K(y_j, x_i).

The symmetry of this inner product follows from the symmetry of K and the non-degeneracy follows from the fact that K is positive definite.

Let H be the completion of H0 with respect to this inner product. Then H consists of functions of the form

f(x) = \sum_{i=1}^\infty a_i K_{x_i} (x)

where \sum_{i=1}^\infty a_i^2 K (x_i, x_i) < \infty. The fact that the above sum converges for every x follows from the Cauchy-Schwartz inequality.

Now we can check the RKHS property, (*):

\langle f, K_x \rangle = \left \langle \sum_{i=1}^\infty a_i K_{x_i}, K_x \right \rangle = \sum_{i=1}^\infty a_i K (x_i, x) = f(x).

To prove uniqueness, let G be another Hilbert space of functions for which K is a reproducing kernel. For any x and y in E, (*) implies that \langle K_x, K_y \rangle_H = K(x, y) = \langle K_x, K_y \rangle_G. By linearity, \langle \cdot, \cdot \rangle_H = \langle \cdot, \cdot \rangle_G on the span of \{K_x:\ x \in E \} . Then G = H by the uniqueness of the completion.

[edit] Bergman kernel

The Bergman kernel is defined for open sets D in Cn. Take the Hilbert H space of square-integrable functions, for the Lebesgue measure on D, that are holomorphic functions. The theory is non-trivial in such cases as there are such functions, which are not identically zero. Then H is a reproducing kernel space, with kernel function the Bergman kernel; this example, with n = 1, was introduced by Bergman in 1922.

[edit] See also

[edit] References

  • Nachman Aronszajn, Theory of Reproducing Kernels, Transactions of the American Mathematical Society, volume 68, number 3, pages 337-404, 1950.
  • Felipe Cucker and Steve Smale, On the Mathematical Foundations of Learning, Bulletin of the American Mathematical Society, volume 39, number 1, pages 1-49, 2001.