Gaussian process

In probability theory and statistics, Gaussian processes are a family of stochastic processes. In a Gaussian process, every point in some input space is associated with a normally distributed random variable. Moreover, every finite collection of those random variables has a multivariate normal distribution. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions.

The concept of Gaussian processes is named after Carl Friedrich Gauss because it is based on the notion of the normal distribution which is often called the Gaussian distribution. In fact, Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions.

Gaussian processes are important in statistical modelling because of properties inherited from the normal. For example, if a random process is modeled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times.

Definition

A Gaussian process is a stochastic process X_t, t ∈ T, for which any finite linear combination of samples has a joint Gaussian distribution. More accurately, any linear functional applied to the sample function X_t will give a normally distributed result. Notation-wise, one can write X ~ GP(m,K), meaning the random function X is distributed as a GP with mean function m and covariance function K.^[1] When the input vector t is two- or multi-dimensional, a Gaussian process might be also known as a Gaussian random field.^[2]

Some authors^[3] assume the random variables X_t have mean zero; this greatly simplifies calculations without loss of generality and allows the mean square properties of the process to be entirely determined by the covariance function K.^[4]

Alternative definitions

Alternatively, a process is Gaussian if and only if for every finite set of indices $t_1,\ldots,t_k$ in the index set $T$

{\mathbf{X}}_{t_1, \ldots, t_k} = (\mathbf{X}_{t_1}, \ldots, \mathbf{X}_{t_k})

is a multivariate Gaussian random variable. Using characteristic functions of random variables, the Gaussian property can be formulated as follows: $\left\{X_t ; t\in T\right\}$ is Gaussian if and only if, for every finite set of indices $t_1,\ldots,t_k$ , there are real valued $\sigma_{\ell j}$ , $\mu_\ell$ with $\sigma_{ii} > 0$ such that

\operatorname{E}\left(\exp\left(i \ \sum_{\ell=1}^k s_\ell \ \mathbf{X}_{t_\ell}\right)\right) = \exp \left(-\frac{1}{2} \, \sum_{\ell, j} \sigma_{\ell j} s_\ell s_j + i \sum_\ell \mu_\ell s_\ell\right).

The numbers $\sigma_{\ell j}$ and $\mu_\ell$ can be shown to be the covariances and means of the variables in the process.^[5]

Covariance functions

A key fact of Gaussian processes is that they can be completely defined by their second-order statistics.^[2] Thus, if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. The covariance matrix K between all the pair of points x and x' specifies a distribution on functions and is known as the Gram matrix. Importantly, because every valid covariance function is a scalar product of vectors, by construction the matrix K is a non-negative definite matrix. Equivalently, the covariance function K is a non-negative definite function in the sense that for every pair x and x', K(x,x') ≥ 0; if K(,) > 0 then K is called positive definite. Importantly the non-negative definiteness of K enables its spectral decomposition using the Karhunen–Loeve expansion. Basic aspects that can be defined through the covariance function are the process' stationarity, isotropy, smoothness and periodicity.^[6]^[7]

Stationarity refers to the process' behaviour regarding the separation of any two points x and x' . If the process is stationary, it depends on their separation, x − x', while if non-stationary it depends on the actual position of the points x and x'; an example of a stationary process is the Ornstein–Uhlenbeck process. On the contrary, the special case of an Ornstein–Uhlenbeck process, a Brownian motion process, is non-stationary.

If the process depends only on |x − x'|, the Euclidean distance (not the direction) between x and x' then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous;^[8] in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function.^[6] If we expect that for "near-by" input points x and x' their corresponding output points y and y' to be "near-by" also, then the assumption of smoothness is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornstein–Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input x to a two dimensional vector u(x) = (cos(x), sin(x)).

Usual covariance functions

There are a number of common covariance functions:^[7]

Constant : $K_\text{C}(x,x') = C$
Linear: $K_\text{L}(x,x') = x^T x'$
Gaussian Noise: $K_\text{GN}(x,x') = \sigma^2 \delta_{x,x'}$
Squared Exponential: $K_\text{SE}(x,x') = \exp \Big(-\frac{|d|^2}{2l^2} \Big)$
Ornstein–Uhlenbeck: $K_\text{OU}(x,x') = \exp \Big(-\frac{|d| }{l} \Big)$
Matérn: $K_\text{Matern}(x,x') = \frac{2^{1-\nu}}{\Gamma(\nu)} \Big(\frac{\sqrt{2\nu}|d|}{l} \Big)^\nu K_{\nu}\Big(\frac{\sqrt{2\nu}|d|}{l} \Big)$
Periodic: $K_\text{P}(x,x') = \exp\Big(-\frac{ 2\sin^2(\frac{d}{2})}{ l^2} \Big)$
Rational Quadratic: $K_\text{RQ}(x,x') = (1+|d|^2)^{-\alpha}, \quad \alpha \geq 0$

Here $d = x- x'$ . The parameter $l$ is the characteristic length-scale of the process (practically, "how far apart" two points $x$ and $x'$ have to be for $X$ to change significantly), δ is the Kronecker delta and σ the standard deviation of the noise fluctuations. Here $K_\nu$ is the modified Bessel function of order $\nu$ and $\Gamma$ is the gamma function evaluated for $\nu$ . Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

Clearly, the inferential results are dependent on the values of the hyperparameters θ (e.g. $l$ and σ) defining the model's behaviour. A popular choice for θ is to provide maximum a posteriori (MAP) estimates of it by maximizing the marginal likelihood of the process; the marginalization being done over the observed process values $y$ .^[7] This approach is also known as maximum likelihood II, evidence maximization, or Empirical Bayes.^[4]

Brownian Motion as the Integral of Gaussian processes

A Wiener process (aka brownian motion) is the integral of a white noise Gaussian process. It is not stationary, but it has stationary increments.

The Ornstein–Uhlenbeck process is a stationary Gaussian process.

The Brownian bridge is the integral of a Gaussian process whose increments are not independent.

The fractional Brownian motion is the integral of a Gaussian process whose covariance function is a generalisation of Wiener process.

Applications

A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference.^[7]^[9] Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian.

Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging.^[10] Gaussian processes are thus useful as a powerful non-linear multivariate interpolation tool. Additionally, Gaussian process regression can be extended to address learning tasks in both supervised (e.g. probabilistic classification^[7]) and unsupervised (e.g. manifold learning^[2]) learning frameworks.

Gaussian process prediction

When concerned with a general Gaussian process regression problem, it is assumed that for a Gaussian process f observed at coordinates x, the vector of values f(x) is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates |x|. Therefore under the assumption of a zero-mean distribution, f (x) ∼ N (0, K(θ,x,x')), where K(θ,x,x') is the covariance matrix between all possible pairs (x,x') for a given set of hyperparameters θ.^[7] As such the log marginal likelihood is:

\log p(f(x)|\theta,x) = -\frac{1}{2}f(x)^T K(\theta,x,x')^{-1} f(x) -\frac{1}{2} \log \det(K(\theta,x,x')) - \frac{|x|}{2} \log 2\pi

and maximizing this marginal likelihood towards θ provides the complete specification of the Gaussian process f. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified θ making predictions about unobserved values f(x*) at coordinates x* is then only a matter of drawing samples from the predictive distribution p(y*|x*,f(x),x) = N(y*|A,B) where the posterior mean estimate A is defined as:

A = K(\theta,x^*,x) K(\theta,x,x')^{-1} f(x)

and the posterior variance estimate B is defined as:

B = K(\theta,x^*,x^*) - K(\theta,x^*,x) K(\theta,x,x')^{-1} K(\theta,x^*,x)^T

where K(θ,x*,x) is the covariance of between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector θ, K(θ,x,x') and f(x) are defined as before and K(θ,x*,x*) is the variance at point x* as dictated by θ. It is important to note that practically the posterior mean estimate f(x*) (the "point estimate") is just a linear combination of the observations f(x); in a similar manner the variance of f(x*) is actually independent of the observations f(x). A known bottleneck in Gaussian process prediction is that the computational complexity of prediction is cubic in the number of points |x| and as such can become unfeasible for larger data sets.^[6] Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue.^[11]^[12]

Notes

↑ Rasmussen, C. E. (2004). "Gaussian Processes in Machine Learning". Advanced Lectures on Machine Learning. Lecture Notes in Computer Science 3176. pp. 63–71. doi:10.1007/978-3-540-28650-9_4. ISBN 978-3-540-23122-6.
↑ 2.0 2.1 2.2 Bishop, C.M. (2006). Pattern Recognition and Machine Learning. Springer. ISBN 0-387-31073-8.
↑ Simon, Barry (1979). Functional Integration and Quantum Physics. Academic Press.
↑ 4.0 4.1 Seeger, Matthias (2004). "Gaussian Processes for Machine Learning". International Journal of Neural Systems 14 (2): 69–104. doi:10.1142/s0129065704001899.
↑ Dudley, R.M. (1989). Real Analysis and Probability. Wadsworth and Brooks/Cole.
↑ 6.0 6.1 6.2 Barber, David (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press. ISBN 978-0-521-51814-7.
↑ 7.0 7.1 7.2 7.3 7.4 7.5 Rasmussen, C.E.; Williams, C.K.I (2006). Gaussian Processes for Machine Learning. MIT Press. ISBN 0-262-18253-X.
↑ Grimmett, Geoffrey; David Stirzaker (2001). Probability and Random Processes. Oxford University Press. ISBN 0198572220.
↑ Liu, W.; Principe, J.C. and Haykin, S. (2010). Kernel Adaptive Filtering: A Comprehensive Introduction. John Wiley. ISBN 0-470-44753-2.
↑ Stein, M.L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer.
↑ Smola, A.J.; Schoellkopf, B. (2000). "Sparse greedy matrix approximation for machine learning". Proceedings of the Seventeenth International Conference on Machine Learning: 911–918.
↑ Csato, L.; Opper, M. (2002). "Sparse on-line Gaussian processes". Neural Computation 14 (3): 641–668. doi:10.1162/089976602317250933.

External links

Software

ooDACE - A flexible object-oriented Kriging matlab toolbox.
GPstuff - Gaussian process toolbox for Matlab and Octave
GPy - A Gaussian processes framework in Python
Interactive Gaussian process regression demo
Basic Gaussian process library written in C++11

Video tutorials

Stochastic processes

Discrete time	Bernoulli process Branching process Chinese restaurant process Galton–Watson process Independent and identically distributed random variables Markov chain Moran process Random walk Loop-erased Self-avoiding

Continuous time	Bessel process Birth–death process Brownian motion Bridge Excursion Fractional Geometric Meander Cauchy process Contact process Continuous-time random walk Cox process Diffusion process Empirical process Feller process Fleming–Viot process Gamma process Hunt process Interacting particle systems Itō diffusion Itō process Jump diffusion Jump process Lévy process Local time Markov additive process McKean–Vlasov process Ornstein–Uhlenbeck process Poisson process Compound Non-homogeneous Point process Schramm–Loewner evolution Semimartingale Sigma-martingale Stable process Superprocess Telegraph process Variance gamma process Wiener process Wiener sausage

Both	Branching process Gaussian process Hidden Markov model (HMM) Markov process Martingale Differences Local Sub- Super- Random dynamical system Regenerative process Renewal process White noise

Fields and other	Dirichlet process Gaussian random field Gibbs measure Hopfield model Ising model Potts model Boolean network Markov random field Percolation Pitman–Yor process Point process Cox Poisson Random field Random graph

Time series models	Autoregressive conditional heteroskedasticity (ARCH) model Autoregressive integrated moving average (ARIMA) model Autoregressive (AR) model Autoregressive–moving-average (ARMA) model Generalized autoregressive conditional heteroskedasticity (GARCH) model Moving-average (MA) model

Financial models	Black–Derman–Toy Black–Karasinski Black–Scholes Chen Constant elasticity of variance (CEV) Cox–Ingersoll–Ross (CIR) Garman–Kohlhagen Heath–Jarrow–Morton (HJM) Heston Ho–Lee Hull–White LIBOR market SABR volatility Vašíček

Actuarial models	Bühlmann Cramér–Lundberg Risk process Sparre–Anderson

Queueing models	Bulk Fluid Generalized queueing network M/G/1 M/M/1 M/M/c

Properties	Càdlàg paths Continuous Continuous paths Ergodic Exchangeable Feller-continuous Gauss–Markov Markov Mixing Piecewise deterministic Predictable Progressively measurable Self-similar Stationary Time-reversible

Limit theorems	Central limit theorem Donsker's theorem Doob's martingale convergence theorems Ergodic theorem Fisher–Tippett–Gnedenko theorem Large deviation principle Law of large numbers (weak/strong) Law of the iterated logarithm Maximal ergodic theorem Sanov's theorem

Inequalities	Burkholder–Davis–Gundy Doob's martingale Kunita–Watanabe

Tools	Cameron–Martin formula Convergence of random variables Doléans-Dade exponential Doob decomposition theorem Doob–Meyer decomposition theorem Doob's optional stopping theorem Dynkin's formula Feynman–Kac formula Filtration Girsanov theorem Infinitesimal generator Itō integral Itō's lemma Kolmogorov continuity theorem Kolmogorov extension theorem Lévy–Prokhorov metric Malliavin calculus Martingale representation theorem Optional stopping theorem Prohorov theorem Quadratic variation Reflection principle Skorokhod integral Skorokhod's representation theorem Skorokhod space Snell envelope Stochastic differential equation Tanaka Stopping time Stratonovich integral Uniform integrability Usual hypotheses Wiener space Classical Abstract

Disciplines	Actuarial mathematics Econometrics Ergodic theory Extreme value theory (EVT) Large deviations theory Mathematical finance Mathematical statistics Probability theory Queueing theory Renewal theory Ruin theory Statistics Stochastic analysis Time series analysis Machine learning

List of topics Category